SeedVR<T>

Models & Types · AiDotNet.Video.Enhancement

SeedVR: seeding infinity in diffusion transformer towards generic video restoration, as the authors released it.

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoSuperResolutionBase<T> → SeedVR<T>

For Beginners

SeedVR fixes degraded video (blur, noise, compression, low resolution) with a large generative model. It looks at the video in big overlapping tiles so it can handle any size, and uses a learned picture of what clean video looks like to fill in lost detail.

How It Works

SeedVR (Wang et al. 2025) restores video with a rectified-flow MM-DiT whose attention runs in large windows: each block cuts the latent token grid into windows (a fixed count at 720p, so any resolution works), every window’s video tokens attend to themselves and the prompt with a window-local 3D rotary embedding, and alternate blocks shift the windows by half. Video and text keep separate weights throughout. The low-quality clip enters as its causal-VAE latent, lightly noised and concatenated channel-wise with the noisy latent and a mask channel.

Sampling follows the release: 50 Euler steps over the uniform trailing timesteps with velocity (v = ε − x₀) prediction, classifier-free guidance between the precomputed positive and negative prompt embeddings, and the clean estimate at the last step. Training is SD3’s flow matching, which the paper follows: logit-normal timesteps shifted with the resolution (the release’s timestep_transform), the MSE on the velocity, the condition noised at a quarter of the schedule.

The prompt embeddings are part of the release (text-encoder outputs computed once) and are held frozen; untrained, they are zero. The input is upscaled bicubically by ScaleFactor in place of the release’s resize to the 720p area, clips are padded to 4k + 1 frames and to multiples of 16 pixels, and the output is cropped back.

Constructors

Constructor Summary
SeedVR(NeuralNetworkArchitecture<T> architecture, SeedVROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a SeedVR model in native training mode.
SeedVR(NeuralNetworkArchitecture<T> architecture, string modelPath, SeedVROptions options) Creates a SeedVR model in native training mode.

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape) Frames are pixels on the 0-255 scale; the network divides them by 255.
LayerInputDomain GetOutputDomain(int[] outputShape) Restored frames come back on the same 0-255 scale.
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> Upscale(Tensor<T> lowResFrames)