RealisVSR<T>

Models & Types · AiDotNet.Video.Enhancement

RealisVSR: detail-enhanced diffusion for real-world 4K video super-resolution, a Consistency-Preserved ControlNet on Wan 2.1.

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoSuperResolutionBase<T> → RealisVSR<T>

For Beginners

RealisVSR takes a low-quality video and uses a large pretrained video generator to produce a sharp, high-resolution version. A small side network reads the low-quality frames and steers the generator, and its training loss puts extra weight on fine detail such as text and texture so those are not smoothed away.

How It Works

RealisVSR (Zhao et al. 2025) keeps Wan 2.1-1.3B frozen and conditions it on the low-resolution clip through a ControlNet branch that sees only the condition: the Wan VAE latents of the bicubic upscale, patch-embedded and passed through a few Wan blocks that share the main network’s prompt, time modulation and rotary positions. After main block i, the branch’s feature ⌊i / r⌋ scaled by a learned γ is added (Eq. 1). Sampling integrates the rectified flow from noise with Euler steps.

Training (Sec. 3.3) updates only the branch with the High-Frequency Rectified Diffusion Loss: the flow-matching MSE on the velocity plus a weighted Haar wavelet loss (Eq. 4) and a HOG loss (Eq. 5) on the same velocity, so edges and textures weigh more than the flat low-frequency content the MSE is dominated by.

The authors’ repository is no longer published, so the unstated choices follow the paper’s backbone and the ControlNet convention it builds on; each is documented on RealisVSROptions. The branch’s blocks start as copies of the main network’s first blocks and its patch embedding is new. Clips are padded to 4k + 1 frames and to sides the VAE and patch grid divide, and the output is cropped back, so every input frame and pixel has an output.

Constructors

Constructor Summary
RealisVSR(NeuralNetworkArchitecture<T> architecture, RealisVSROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a RealisVSR model in native training mode.
RealisVSR(NeuralNetworkArchitecture<T> architecture, string modelPath, RealisVSROptions options) Creates a RealisVSR model in native training mode.

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape) Frames are pixels on the 0-255 scale; the network divides them by 255.
LayerInputDomain GetOutputDomain(int[] outputShape) Upscaled frames come back on the same 0-255 scale.
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> Upscale(Tensor<T> lowResFrames)