MGLDVSR<T>
Models & Types · AiDotNet.Video.Enhancement
MGLD-VSR: motion-guided latent diffusion for temporally consistent real-world video super-resolution.
Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoSuperResolutionBase<T> → MGLDVSR<T>
For Beginners
Diffusion models make images by removing noise step by step. MGLD-VSR uses optical flow to keep neighbouring frames consistent while it removes the noise, and a decoder that also looks at neighbouring frames, so the upscaled video does not flicker.
How It Works
MGLD-VSR (Yang et al., ECCV 2024) builds on StableSR: a frozen Stable Diffusion 2.1 UNet whose residual blocks are SPADE-modulated by a time-aware encoding of the LR latent, with temporal convolutions and temporal attention added to its middle block over five-frame clips. Sampling starts from the LR latent noised to the last timestep and, after every DDPM step, moves the latents down the gradient of an occlusion-masked warping error between neighbouring latents (flows from RAFT on the LR frames). A video decoder (SD’s KL decoder with CFW modules fusing the LQ encoder’s features and temporal convolutions) is trained afterwards with L1, LPIPS, frame-difference, edge-weighted warping and adversarial losses.
Constructors
| Constructor | Summary |
|---|---|
MGLDVSR(NeuralNetworkArchitecture<T> architecture, MGLDVSROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates an MGLD-VSR model in native training mode. |
MGLDVSR(NeuralNetworkArchitecture<T> architecture, string modelPath, MGLDVSROptions options) |
Creates an MGLD-VSR model in native training mode. |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
Frames are pixels on the 0-255 scale; the network divides them by 255. |
LayerInputDomain GetOutputDomain(int[] outputShape) |
Upscaled frames come back on the same 0-255 scale. |
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Tensor<T> Upscale(Tensor<T> lowResFrames) |