MGLDVSR<T>

Models & Types · AiDotNet.Video.Enhancement

MGLD-VSR: motion-guided latent diffusion for temporally consistent real-world video super-resolution.

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoSuperResolutionBase<T> → MGLDVSR<T>

For Beginners

Diffusion models make images by removing noise step by step. MGLD-VSR uses optical flow to keep neighbouring frames consistent while it removes the noise, and a decoder that also looks at neighbouring frames, so the upscaled video does not flicker.

How It Works

MGLD-VSR (Yang et al., ECCV 2024) builds on StableSR: a frozen Stable Diffusion 2.1 UNet whose residual blocks are SPADE-modulated by a time-aware encoding of the LR latent, with temporal convolutions and temporal attention added to its middle block over five-frame clips. Sampling starts from the LR latent noised to the last timestep and, after every DDPM step, moves the latents down the gradient of an occlusion-masked warping error between neighbouring latents (flows from RAFT on the LR frames). A video decoder (SD’s KL decoder with CFW modules fusing the LQ encoder’s features and temporal convolutions) is trained afterwards with L1, LPIPS, frame-difference, edge-weighted warping and adversarial losses.

Constructors

Constructor Summary
MGLDVSR(NeuralNetworkArchitecture<T> architecture, MGLDVSROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates an MGLD-VSR model in native training mode.
MGLDVSR(NeuralNetworkArchitecture<T> architecture, string modelPath, MGLDVSROptions options) Creates an MGLD-VSR model in native training mode.

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape) Frames are pixels on the 0-255 scale; the network divides them by 255.
LayerInputDomain GetOutputDomain(int[] outputShape) Upscaled frames come back on the same 0-255 scale.
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> Upscale(Tensor<T> lowResFrames)