FlashVSR<T>
Models & Types · AiDotNet.Video.Enhancement
FlashVSR: real-time diffusion-based streaming video super-resolution, distilled to one step from Wan 2.1.
Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoSuperResolutionBase<T> → FlashVSR<T>
For Beginners
FlashVSR sharpens video in real time. It starts from a large video generation model, teaches it to produce a sharp frame in a single pass, and limits each pass to recent frames and nearby regions so it keeps up with a live stream.
How It Works
FlashVSR (Zhuang et al. 2025) conditions Wan 2.1-1.3B on the low-resolution clip through a light projection whose tokens are added to the DiT’s, not through the VAE encoder. One step at timestep 1000 maps noise to the latent (z = ε − v), and the tiny conditional decoder turns it into frames using the LR clip as well. The release streams two latent frames at a time against a cache of the previous three chunks, with attention restricted to the top-scoring (2, 8, 8)-token blocks inside a locality window; evaluated over the whole clip with the equivalent block-causal mask, as here, the result is the same. Colors are matched to the LR upscale per frame (AdaIN).
Training follows Sec. 3-4 in four stages: the tiny decoder alone (L2 and LPIPS against the ground truth and against Wan’s decoder); flow matching with full attention, which becomes the frozen teacher; flow matching with the sparse causal attention; and distribution-matching distillation of the one-step model against the teacher and a fake-score critic, with L2 and LPIPS on the decoded frames. The DiT trains through LoRA adapters, the LR projection fully.
Frames are padded rather than dropped: the release repeats the last frame four times and keeps the largest 8n + 1 frames, cropping the rest, and crops each side to a multiple of 128; here the clip is padded to the next valid length and size and the output cropped back, so every input frame and pixel has an output. The fixed prompt’s T5 embedding is part of the checkpoint and held frozen; untrained, it is zero. The image half of the paper’s Stage 1 data is left out: the released LR projection emits no token for a lone frame.
Constructors
| Constructor | Summary |
|---|---|
FlashVSR(NeuralNetworkArchitecture<T> architecture, FlashVSROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a FlashVSR model in native training mode. |
FlashVSR(NeuralNetworkArchitecture<T> architecture, string modelPath, FlashVSROptions options) |
Creates a FlashVSR model in native training mode. |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
Frames are pixels on the 0-255 scale; the network divides them by 255. |
LayerInputDomain GetOutputDomain(int[] outputShape) |
Upscaled frames come back on the same 0-255 scale. |
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Tensor<T> Upscale(Tensor<T> lowResFrames) |