RealESRGANVideo<T>

Models & Types · AiDotNet.Video.Enhancement

Real-ESRGAN Video: Real-ESRGAN real-world super-resolution applied to every frame of a video.

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoSuperResolutionBase<T> → RealESRGANVideo<T>

For Beginners

Real-ESRGAN is one of the most widely-used practical upscaling tools. It learns to handle all the messy artifacts (compression, noise, blur) that real videos have, not just the simple downscaling used in lab benchmarks, and upscales each frame of a video by 4x.

How It Works

Wang et al. 2021 disclose no video network: the authors’ release upscales video with inference_realesrgan_video.py, which runs the image model on each frame independently. This class is that pipeline with the paper’s generator, RRDBNet (64 features, 23 residual-in-residual dense blocks, growth 32, x4), trained as RealESRGAN_x4plus (options/train_realesrgan_x4plus.yml): an L1 loss and a VGG19 perceptual loss (conv1_2 .. conv5_4 before ReLU, weights 0.1, 0.1, 1, 1, 1) against the unsharp-masked target and a vanilla GAN loss weighted 0.1 from a spectrally normalized U-Net discriminator; generator and discriminator alternate with Adam (0.9, 0.99) at 1e-4, and inference uses a 0.999 moving average of the generator.

The release synthesizes training pairs with its second-order degradation pipeline and starts this stage from the L1-trained Real-ESRNet and an ImageNet-pretrained VGG19; here the caller supplies degraded/clean pairs and the networks start from the release’s initializations.

Example

var arch = new NeuralNetworkArchitecture<float>(inputType: InputType.FourDimensional,
    taskType: NeuralNetworkTaskType.Regression, inputFrames: 4, inputHeight: 64, inputWidth: 64, inputDepth: 3);
var model = new RealESRGANVideo<float>(arch, new RealESRGANVideoOptions());
model.Train(degradedFrames, cleanFrames);     // [4, 3, 64, 64] -> [4, 3, 256, 256], pixel values 0-255
var upscaled = model.Upscale(degradedFrames);

Constructors

Constructor Summary
RealESRGANVideo(NeuralNetworkArchitecture<T> architecture, RealESRGANVideoOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a Real-ESRGAN Video model in native training mode.
RealESRGANVideo(NeuralNetworkArchitecture<T> architecture, string modelPath, RealESRGANVideoOptions options) Creates a Real-ESRGAN Video model in native training mode.

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape) Frames are pixels on the 0-255 scale; the network divides them by 255.
LayerInputDomain GetOutputDomain(int[] outputShape) Upscaled frames come back on the same 0-255 scale.
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> Upscale(Tensor<T> lowResFrames)