DAMVSR<T>

Models & Types · AiDotNet.Video.Enhancement

DAM-VSR: video super-resolution on Stable Video Diffusion that takes appearance from an image super-resolution model and motion from the low-resolution video.

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoSuperResolutionBase<T> → DAMVSR<T>

For Beginners

DAM-VSR splits video upscaling in two. A strong photo upscaler makes one frame sharp and detailed, the low-resolution video says how everything moves, and a video diffusion model generates the remaining frames so they match the sharp frame and follow the motion.

How It Works

The network is Kong et al.’s DAM-VSR (SIGGRAPH 2025, arXiv 2507.01012) on Stable Video Diffusion as diffusers releases it. An image super-resolution model upscales the clip’s first frame; its VAE latent, repeated over the frames, is concatenated with the noisy latent, and its CLIP ViT-H/14 embedding is the U-Net’s cross-attention context. A video ControlNet, a copy of the U-Net’s encoder with its temporal layers, sees the same input plus the low-resolution video through ControlNet’s conditioning embedding, and its zero-initialized outputs are added to the U-Net’s skips and middle block. The denoiser is SVD’s UNetSpatioTemporalConditionModel with EDM v-prediction preconditioning.

Sampling enters a 30-step Euler schedule 60% of the way down from the noised latent of the bicubically upscaled clip (SDEdit). With motion-aligned bidirectional sampling each step also runs the clip backwards from its super-resolved last frame, reusing the forward pass’s temporal attention maps rotated 180 degrees with the backward pass’s own temporal value and output projections, and averages the two predictions. Training is the paper’s three stages: the ControlNet, then the backward projections, then the VAE decoder on L2, perceptual and GAN losses of 8-step samples. The paper releases no code. Where it is silent this follows SVD’s pipeline: added time ids (fps - 1, motion bucket 127, noise augmentation 0.02), unscaled reference latents, EDM training noise log σ ~ N(0.7, 1.6²), the ground-truth first frame as the training reference, no classifier-free guidance (the low-resolution video conditions every frame), and RealBasicVSR’s discriminator. The decoder is fine-tuned directly rather than through LoRA adapters, and the image super-resolution model is supplied by the caller (ReferenceUpscaler), bicubic when none is.

Example

var arch = new NeuralNetworkArchitecture<float>(inputType: InputType.FourDimensional,
    taskType: NeuralNetworkTaskType.Regression, inputFrames: 14, inputHeight: 64, inputWidth: 64, inputDepth: 3);
var model = new DAMVSR<float>(arch, new DAMVSROptions());
model.ReferenceUpscaler = frame => myImageSuperResolver.Upscale(frame);
var upscaled = model.Upscale(lowResClip);      // [14, 3, 64, 64] -> [14, 3, 256, 256], pixel values 0-255

Constructors

Constructor Summary
DAMVSR(NeuralNetworkArchitecture<T> architecture, DAMVSROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a DAM-VSR model in native training mode.
DAMVSR(NeuralNetworkArchitecture<T> architecture, string modelPath, DAMVSROptions options) Creates a DAM-VSR model in native training mode.

Properties

Property Summary
Func<Tensor<T>, Tensor<T>> ReferenceUpscaler { get; set; } The image super-resolution model that makes the reference frames: [3, h, w] in [0, 1] to [3, s·h, s·w] in [0, 1].

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape) Frames are pixels on the 0-255 scale; the network divides them by 255.
LayerInputDomain GetOutputDomain(int[] outputShape) Upscaled frames come back on the same 0-255 scale.
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> Upscale(Tensor<T> lowResFrames)