IconVSR<T>

Models & Types · AiDotNet.Video.Enhancement

IconVSR: information-aggregation with coupled propagation for video super-resolution.

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoSuperResolutionBase<T> → IconVSR<T>

For Beginners

In long videos, processing frames one-by-one can accumulate errors. IconVSR solves this by picking “keyframes” as reliable reference points and letting the forward/backward passes share information, like having checkpoints in a relay race.

How It Works

The network is the authors’ IconVSRNet (open-mmlab/mmagic, models/editors/iconvsr): BasicVSR’s bidirectional recurrent propagation with SPyNet flow alignment, plus information refill and coupled propagation. At keyframes (every fifth frame and the last) the propagated feature is fused by a 3x3 convolution with features an EDVR-M extractor (PCD deformable alignment and TSA fusion) computes from the five surrounding frames, which stops alignment errors from accumulating over long sequences; the forward branch also takes the backward branch’s feature at each frame.

Training follows configs/iconvsr/iconvsr_2xb4_reds4.py: Charbonnier loss, Adam (0.9, 0.99) at 2e-4 with SPyNet at 0.125 times the rate, SPyNet and EDVR frozen for the first 5,000 iterations, cosine annealing to 1e-7. The release loads SPyNet and EDVR-M pretrained; here they start from their release initializations.

Example

var arch = new NeuralNetworkArchitecture<float>(inputType: InputType.FourDimensional,
    taskType: NeuralNetworkTaskType.Regression, inputFrames: 15, inputHeight: 64, inputWidth: 64, inputDepth: 3);
var model = new IconVSR<float>(arch, new IconVSROptions());
model.Train(lowResClip, highResClip);         // [15, 3, 64, 64] -> [15, 3, 256, 256], pixel values 0-255
var upscaled = model.Upscale(lowResClip);

Constructors

Constructor Summary
IconVSR(NeuralNetworkArchitecture<T> architecture, IconVSROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates an IconVSR model in native training mode.
IconVSR(NeuralNetworkArchitecture<T> architecture, string modelPath, IconVSROptions options) Creates an IconVSR model in native training mode.

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape) Frames are pixels on the 0-255 scale; the network divides them by 255.
LayerInputDomain GetOutputDomain(int[] outputShape) Upscaled frames come back on the same 0-255 scale.
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> Upscale(Tensor<T> lowResFrames)