BasicVSR<T>

Models & Types · AiDotNet.Video.Enhancement

BasicVSR: baseline bidirectional recurrent video super-resolution.

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoSuperResolutionBase<T> → BasicVSR<T>

For Beginners

BasicVSR processes video frames like reading a book forward and backward simultaneously. By looking in both directions, each frame gets context from past AND future frames. Optical flow tells the model how pixels moved between frames, so it can properly align them before combining their information.

How It Works

The network is the authors’ BasicVSRNet (open-mmlab/mmagic, models/editors/basicvsr/basicvsr_net.py). SPyNet estimates the flow between neighbouring low-resolution frames. A backward branch runs from the last frame to the first: the previous state is warped by the flow to the next frame, concatenated with the current frame and refined by a 3x3 convolution and 30 residual blocks. A forward branch does the same from the first frame to the last, and at each frame the two states are fused by a 1x1 convolution and reconstructed by two pixel-shuffle x2 stages onto the bilinearly upsampled frame, so the output is four times the input’s height and width.

Training follows configs/basicvsr/basicvsr_2xb4_reds4.py: Charbonnier loss, Adam (0.9, 0.99) at 2e-4 with SPyNet at 0.125 times the rate and frozen for the first 5,000 iterations, cosine-annealed to 1e-7 over 300,000. The release loads a pretrained SPyNet; this native model starts it from PyTorch’s default initialization.

Example

var arch = new NeuralNetworkArchitecture<float>(inputType: InputType.FourDimensional,
    taskType: NeuralNetworkTaskType.Regression, inputFrames: 15, inputHeight: 64, inputWidth: 64, inputDepth: 3);
var model = new BasicVSR<float>(arch, new BasicVSROptions());
model.Train(lowResClip, highResClip);         // [15, 3, 64, 64] -> [15, 3, 256, 256], pixel values 0-255
var upscaled = model.Upscale(lowResClip);

Constructors

Constructor Summary
BasicVSR(NeuralNetworkArchitecture<T> architecture, BasicVSROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a BasicVSR model in native training mode.
BasicVSR(NeuralNetworkArchitecture<T> architecture, string modelPath, BasicVSROptions options) Creates a BasicVSR model in native training mode.

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape) Frames are pixels on the 0-255 scale; the network divides them by 255.
LayerInputDomain GetOutputDomain(int[] outputShape) Upscaled frames come back on the same 0-255 scale.
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> Upscale(Tensor<T> lowResFrames)