StreamDiffVSR<T>

Models & Types · AiDotNet.Video.Enhancement

Stream-DiffVSR: low-latency streamable video super-resolution by auto-regressive diffusion.

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoSuperResolutionBase<T> → StreamDiffVSR<T>

For Beginners

most video upscalers look at future frames, which adds delay. Stream-DiffVSR only uses frames it has already produced, so it suits live streams and video calls.

How It Works

Stream-DiffVSR (Shiu et al., 2025) upscales a video frame by frame using only the past. Each frame is denoised in four DDIM steps by a UNet distilled from the Stable Diffusion x4 upscaler, which reads the noisy latent next to the LR frame. From the second frame on, the previous HR output is warped to the current frame with RAFT flow (computed on the bicubically upscaled LR frames) and guides the denoiser through a ControlNet (Auto-regressive Temporal Guidance), and the temporal tiny autoencoder’s decoder fuses the encoder’s features of that warped frame (the Temporal Processor Module).

The release trains in three stages: the UNet’s rollout distillation, the TPM, then the ARTG module with the UNet, the autoencoder and RAFT frozen. Train runs the last stage as train/train_artg.py does: the previous frame is generated by the four-step UNet alone, decoded, warped and encoded; the current frame’s four-step rollout runs with the ControlNet; the loss is the final step’s v-prediction MSE plus LPIPS (AlexNet) on the decoded frame, and on every fifth step a PatchGAN discriminator on the frame next to the upscaled LR frame trains and, from the start iteration, adds a non-saturating adversarial term at taming’s adaptive weight. The release’s discriminator scores the generator’s frame in eval mode (running batch-norm statistics); here it always normalizes with the batch’s.

Constructors

Constructor Summary
StreamDiffVSR(NeuralNetworkArchitecture<T> architecture, StreamDiffVSROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a Stream-DiffVSR model in native training mode.
StreamDiffVSR(NeuralNetworkArchitecture<T> architecture, string modelPath, StreamDiffVSROptions options) Creates a Stream-DiffVSR model in native training mode.

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape) Frames are pixels on the 0-255 scale; the network divides them by 255.
LayerInputDomain GetOutputDomain(int[] outputShape) Upscaled frames come back on the same 0-255 scale.
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> Upscale(Tensor<T> lowResFrames)