RealBasicVSRSharp<T>
Models & Types · AiDotNet.Video.Enhancement
RealBasicVSR-Sharp: the adversarial training stage of RealBasicVSR, trading PSNR for perceptual quality.
Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoSuperResolutionBase<T> → RealBasicVSRSharp<T>
For Beginners
There are two ways to measure video quality: mathematical accuracy (PSNR) and visual quality (how it looks to your eyes). The base RealBasicVSR maximizes PSNR, while this “Sharp” variant maximizes visual quality. The Sharp version produces images with crisper textures and more natural-looking details.
How It Works
The generator is the authors’ RealBasicVSRNet, the same network RealBasicVSR trains in the first
stage: an image-cleaning module applied up to three times, then BasicVSR. This class trains the release’s second
stage (configs/real_basicvsr/realbasicvsr_c64b20-1x30x8_8xb1-lr5e-5-150k_reds.py): the generator minimizes the L1
output loss, a VGG19 perceptual loss (conv1_2 .. conv5_4 before ReLU, weights 0.1, 0.1, 1, 1, 1), the L1 cleaning
loss and a vanilla GAN loss weighted 0.05 from a spectrally normalized U-Net discriminator, with Adam (0.9, 0.99) at
5e-5; the discriminator then steps on the real clip and the same, detached output with Adam at 1e-4. The output and
perceptual losses score the unsharp-masked target, the discriminator the unsharpened one. SPyNet stays frozen and
inference uses a 0.999 moving average of the generator.
The release starts this stage from the first stage’s weights and an ImageNet-pretrained VGG19 and SPyNet; here they start from PyTorch’s default initialization (load trained weights through the parameter API), and the caller supplies degraded/clean pairs where the release synthesizes the degradations.
Example
var arch = new NeuralNetworkArchitecture<float>(inputType: InputType.FourDimensional,
taskType: NeuralNetworkTaskType.Regression, inputFrames: 15, inputHeight: 64, inputWidth: 64, inputDepth: 3);
var model = new RealBasicVSRSharp<float>(arch, new RealBasicVSRSharpOptions());
model.Train(degradedClip, cleanClip); // [15, 3, 64, 64] -> [15, 3, 256, 256], pixel values 0-255
var upscaled = model.Upscale(degradedClip);
Constructors
| Constructor | Summary |
|---|---|
RealBasicVSRSharp(NeuralNetworkArchitecture<T> architecture, RealBasicVSRSharpOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a RealBasicVSR-Sharp model in native training mode. |
RealBasicVSRSharp(NeuralNetworkArchitecture<T> architecture, string modelPath, RealBasicVSRSharpOptions options) |
Creates a RealBasicVSR-Sharp model in native training mode. |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
Frames are pixels on the 0-255 scale; the network divides them by 255. |
LayerInputDomain GetOutputDomain(int[] outputShape) |
Upscaled frames come back on the same 0-255 scale. |
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Tensor<T> Upscale(Tensor<T> lowResFrames) |