IART<T>

Models & Types · AiDotNet.Video.Enhancement

IART: a recurrent video super-resolution transformer whose frame alignment is learned implicit resampling.

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoSuperResolutionBase<T> → IART<T>

For Beginners

To reuse detail from the previous frame, most video upscalers follow the motion and blend the four nearest pixels with fixed weights, which blurs fine texture. IART lets each pixel look at those four neighbours and decide for itself how much to take from each.

How It Works

The network is the authors’ IART for x4 video super-resolution (Xu et al., “Enhancing Video Super-Resolution via Implicit Resampling-based Alignment”, CVPR 2024; kai422/IART, archs/iart_arch.py and archs/implicit_alignment.py). It is PSRT-recurrent’s network with the alignment replaced: SPyNet flow, a 3x3 embedding to 120 channels, and four recurrent passes (backward, forward, backward, forward), each stacking the aligned previous and second-previous outputs around the current features and running the stack through a SwinIRFM (three RSTBs of six window-attention layers over 3x8x8 windows). Alignment is one shared implicit warp: every current pixel attends, with a single query, over the 2x2 propagated pixels around the point its flow lands on, with sine codes of the window position and of the sub-pixel offset, so the resampling kernel is learned instead of bilinear. The last pass is reconstructed with x4 pixel-shuffle upsampling and added to the bilinearly upsampled input.

Training follows options/IART_REDS_N16_600K.yml: Charbonnier loss (eps 1e-12 under the square root), Adam (0.9, 0.99) at 2e-4 with SPyNet at 0.125 times the rate and frozen for the first 5,000 iterations, cosine annealing to 1e-7 over 600,000 iterations, 16-frame sequences, stochastic depth up to 0.1, and an exponential moving average of the weights (decay 0.999) used for inference. SPyNet is loaded pretrained in the release and starts from PyTorch’s default initialization here. The release’s GELU is the erf form; the library’s is the tanh approximation.

Example

var arch = new NeuralNetworkArchitecture<float>(inputType: InputType.FourDimensional,
    taskType: NeuralNetworkTaskType.Regression, inputFrames: 8, inputHeight: 64, inputWidth: 64, inputDepth: 3);
var model = new IART<float>(arch, new IARTOptions());
model.Train(lowResClip, highResClip);         // [8, 3, 64, 64] -> [8, 3, 256, 256], pixel values 0-255
var upscaled = model.Upscale(lowResClip);

Constructors

Constructor Summary
IART(NeuralNetworkArchitecture<T> architecture, IARTOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates an IART model in native training mode.
IART(NeuralNetworkArchitecture<T> architecture, string modelPath, IARTOptions options) Creates an IART model in native training mode.

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape) Frames are pixels on the 0-255 scale; the network divides them by 255.
LayerInputDomain GetOutputDomain(int[] outputShape) Upscaled frames come back on the same 0-255 scale.
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> Upscale(Tensor<T> lowResFrames)