RealBasicVSR<T>

Models & Types · AiDotNet.Video.Enhancement

RealBasicVSR: real-world video super-resolution with an image-cleaning module.

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoSuperResolutionBase<T> → RealBasicVSR<T>

For Beginners

Real videos from phones or old cameras have noise, blur, and compression artifacts. RealBasicVSR first “cleans” each frame to remove these issues, then upscales the cleaned frames. This prevents noise from spreading through the video during the upscaling process.

How It Works

The network is the authors’ RealBasicVSRNet (open-mmlab/mmagic, models/editors/real_basicvsr): an image-cleaning module of 20 residual blocks predicts a residue that is added to every low-quality frame, repeated up to three times while the mean residue is large, and the cleaned frames go through BasicVSR with 20 blocks per propagation branch, so noise and compression artifacts are removed before the recurrence can spread them.

This class trains the release’s first, PSNR-oriented stage (realbasicvsr_wogan): an L1 loss on the output and an L1 “cleaning” loss of the cleaned frames against the 4x area-downsampled target, both against the unsharp-masked target, with Adam (0.9, 0.99) at 1e-4 and SPyNet frozen; inference uses a 0.999 moving average of the weights. RealBasicVSRSharp is the second, adversarial stage. The release degrades the clean clips with random blur, noise, resizing and compression to make training pairs; here the caller supplies the pairs. SPyNet is loaded pretrained in the release and starts from PyTorch’s default initialization here.

Example

var arch = new NeuralNetworkArchitecture<float>(inputType: InputType.FourDimensional,
    taskType: NeuralNetworkTaskType.Regression, inputFrames: 15, inputHeight: 64, inputWidth: 64, inputDepth: 3);
var model = new RealBasicVSR<float>(arch, new RealBasicVSROptions());
model.Train(degradedClip, cleanClip);         // [15, 3, 64, 64] -> [15, 3, 256, 256], pixel values 0-255
var upscaled = model.Upscale(degradedClip);

Constructors

Constructor Summary
RealBasicVSR(NeuralNetworkArchitecture<T> architecture, RealBasicVSROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a RealBasicVSR model in native training mode.
RealBasicVSR(NeuralNetworkArchitecture<T> architecture, string modelPath, RealBasicVSROptions options) Creates a RealBasicVSR model in native training mode.

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape) Frames are pixels on the 0-255 scale; the network divides them by 255.
LayerInputDomain GetOutputDomain(int[] outputShape) Upscaled frames come back on the same 0-255 scale.
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> Upscale(Tensor<T> lowResFrames)