PSRT<T>

Models & Types · AiDotNet.Video.Enhancement

PSRT-recurrent: a recurrent video super-resolution transformer that aligns frames by moving whole patches.

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoSuperResolutionBase<T> → PSRT<T>

For Beginners

PSRT walks through a video forward and backward in time, carrying what it learned from earlier frames. Instead of bending every pixel of the previous frame to match the current one, it shifts whole 8x8 tiles by their average motion, which keeps fine texture intact, and lets attention sort out the rest.

How It Works

The network is the authors’ PSRT-recurrent for x4 video super-resolution (Shi et al., “Rethinking Alignment in Video Super-Resolution Transformers”, NeurIPS 2022; XPixelGroup/RethinkVSRAlignment, psrt_recurrent_arch.py and psrt_sliding_arch.py): SPyNet flow, a 3x3 embedding to 120 channels, and four recurrent passes (backward, forward, backward, forward). For every frame, a pass stacks the previous output and the second-previous output, each moved by patch alignment, around the current features, and runs the stack through a SwinIRFM: LayerNorm, three RSTBs of six window-attention layers over 3x8x8 windows (alternately shifted by half a window), and a convolution on the centre frame added to the centre input. Patch alignment averages the optical flow over each 8x8 patch and moves the whole patch by that offset with nearest-neighbour sampling, rather than warping every pixel. The last pass is reconstructed with x4 pixel-shuffle upsampling and added to the bilinearly upsampled input.

Training follows options/4126_PSRTRecurrent_mix_precision_REDS_600K_N16.yml: Charbonnier loss (eps 1e-12 under the square root), Adam (0.9, 0.99) at 2e-4 with SPyNet at 0.125 times the rate and frozen for the first 5,000 iterations, cosine annealing to 1e-7 over 600,000 iterations, 16-frame sequences, stochastic depth up to 0.1, and an exponential moving average of the weights (decay 0.999) used for inference. SPyNet is loaded pretrained in the release and starts from PyTorch’s default initialization here. The release’s GELU is the erf form; the library’s is the tanh approximation (they differ by under 1e-3).

Example

var arch = new NeuralNetworkArchitecture<float>(inputType: InputType.FourDimensional,
    taskType: NeuralNetworkTaskType.Regression, inputFrames: 8, inputHeight: 64, inputWidth: 64, inputDepth: 3);
var model = new PSRT<float>(arch, new PSRTOptions());
model.Train(lowResClip, highResClip);         // [8, 3, 64, 64] -> [8, 3, 256, 256], pixel values 0-255
var upscaled = model.Upscale(lowResClip);

Constructors

Constructor Summary
PSRT(NeuralNetworkArchitecture<T> architecture, PSRTOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a PSRT model in native training mode.
PSRT(NeuralNetworkArchitecture<T> architecture, string modelPath, PSRTOptions options) Creates a PSRT model in native training mode.

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape) Frames are pixels on the 0-255 scale; the network divides them by 255.
LayerInputDomain GetOutputDomain(int[] outputShape) Upscaled frames come back on the same 0-255 scale.
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> Upscale(Tensor<T> lowResFrames)