FlashVSR<T>

Models & Types · AiDotNet.Video.Enhancement

FlashVSR: real-time diffusion-based streaming video super-resolution, distilled to one step from Wan 2.1.

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoSuperResolutionBase<T> → FlashVSR<T>

For Beginners

FlashVSR sharpens video in real time. It starts from a large video generation model, teaches it to produce a sharp frame in a single pass, and limits each pass to recent frames and nearby regions so it keeps up with a live stream.

How It Works

FlashVSR (Zhuang et al. 2025) conditions Wan 2.1-1.3B on the low-resolution clip through a light projection whose tokens are added to the DiT’s, not through the VAE encoder. One step at timestep 1000 maps noise to the latent (z = ε − v), and the tiny conditional decoder turns it into frames using the LR clip as well. The release streams two latent frames at a time against a cache of the previous three chunks, with attention restricted to the top-scoring (2, 8, 8)-token blocks inside a locality window; evaluated over the whole clip with the equivalent block-causal mask, as here, the result is the same. Colors are matched to the LR upscale per frame (AdaIN).

Training follows Sec. 3-4 in four stages: the tiny decoder alone (L2 and LPIPS against the ground truth and against Wan’s decoder); flow matching with full attention, which becomes the frozen teacher; flow matching with the sparse causal attention; and distribution-matching distillation of the one-step model against the teacher and a fake-score critic, with L2 and LPIPS on the decoded frames. The DiT trains through LoRA adapters, the LR projection fully.

Frames are padded rather than dropped: the release repeats the last frame four times and keeps the largest 8n + 1 frames, cropping the rest, and crops each side to a multiple of 128; here the clip is padded to the next valid length and size and the output cropped back, so every input frame and pixel has an output. The fixed prompt’s T5 embedding is part of the checkpoint and held frozen; untrained, it is zero. The image half of the paper’s Stage 1 data is left out: the released LR projection emits no token for a lone frame.

Constructors

Constructor Summary
FlashVSR(NeuralNetworkArchitecture<T> architecture, FlashVSROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a FlashVSR model in native training mode.
FlashVSR(NeuralNetworkArchitecture<T> architecture, string modelPath, FlashVSROptions options) Creates a FlashVSR model in native training mode.

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape) Frames are pixels on the 0-255 scale; the network divides them by 255.
LayerInputDomain GetOutputDomain(int[] outputShape) Upscaled frames come back on the same 0-255 scale.
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> Upscale(Tensor<T> lowResFrames)