BasicVSR<T>
Models & Types · AiDotNet.Video.Enhancement
BasicVSR: baseline bidirectional recurrent video super-resolution.
Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoSuperResolutionBase<T> → BasicVSR<T>
For Beginners
BasicVSR processes video frames like reading a book forward and backward simultaneously. By looking in both directions, each frame gets context from past AND future frames. Optical flow tells the model how pixels moved between frames, so it can properly align them before combining their information.
How It Works
The network is the authors’ BasicVSRNet (open-mmlab/mmagic, models/editors/basicvsr/basicvsr_net.py). SPyNet estimates the flow between neighbouring low-resolution frames. A backward branch runs from the last frame to the first: the previous state is warped by the flow to the next frame, concatenated with the current frame and refined by a 3x3 convolution and 30 residual blocks. A forward branch does the same from the first frame to the last, and at each frame the two states are fused by a 1x1 convolution and reconstructed by two pixel-shuffle x2 stages onto the bilinearly upsampled frame, so the output is four times the input’s height and width.
Training follows configs/basicvsr/basicvsr_2xb4_reds4.py: Charbonnier loss, Adam (0.9, 0.99) at 2e-4 with SPyNet at 0.125 times the rate and frozen for the first 5,000 iterations, cosine-annealed to 1e-7 over 300,000. The release loads a pretrained SPyNet; this native model starts it from PyTorch’s default initialization.
Example
var arch = new NeuralNetworkArchitecture<float>(inputType: InputType.FourDimensional,
taskType: NeuralNetworkTaskType.Regression, inputFrames: 15, inputHeight: 64, inputWidth: 64, inputDepth: 3);
var model = new BasicVSR<float>(arch, new BasicVSROptions());
model.Train(lowResClip, highResClip); // [15, 3, 64, 64] -> [15, 3, 256, 256], pixel values 0-255
var upscaled = model.Upscale(lowResClip);
Constructors
| Constructor | Summary |
|---|---|
BasicVSR(NeuralNetworkArchitecture<T> architecture, BasicVSROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a BasicVSR model in native training mode. |
BasicVSR(NeuralNetworkArchitecture<T> architecture, string modelPath, BasicVSROptions options) |
Creates a BasicVSR model in native training mode. |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
Frames are pixels on the 0-255 scale; the network divides them by 255. |
LayerInputDomain GetOutputDomain(int[] outputShape) |
Upscaled frames come back on the same 0-255 scale. |
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Tensor<T> Upscale(Tensor<T> lowResFrames) |