RVRT<T>

Models & Types · AiDotNet.Video.Enhancement

RVRT: recurrent video restoration transformer with guided deformable attention.

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoSuperResolutionBase<T> → RVRT<T>

For Beginners

RVRT works through a video two frames at a time, carrying what it learned forward and then backward. To match each pair with the last one, it follows the estimated motion and then searches a small area around each moved point for the best detail to borrow.

How It Works

The network is the authors’ RVRT for x4 video super-resolution (Liang et al., NeurIPS 2022; JingyunLiang/RVRT, models/network_rvrt.py): SPyNet flow, an RSTB feature extractor over 1x8x8 windows, and four recurrent passes (backward, forward, backward, forward) over two-frame clips. Each pass aligns the propagated clip to the current one with guided deformable attention (offsets are residues on top of composed optical flows, and each query attends over a deformable 3x3 window in both frames of the clip), then refines it with RSTB groups over 2x8x8 windows fed by every earlier pass. An RSTB reconstruction and x4 pixel-shuffle upsampling add to the bilinearly upsampled input.

Training follows KAIR options/rvrt/001_train_rvrt_videosr_bi_reds_30frames.json: Charbonnier loss, Adam (0.9, 0.99) at 4e-4 with SPyNet at 0.25 times the rate and frozen for the first 30,000 iterations, cosine annealing to 1e-7 over 600,000 iterations. SPyNet is loaded pretrained in the release and starts from PyTorch’s default initialization here. The release’s GELU is the erf form; the library’s is the tanh approximation (they differ by under 1e-3).

Example

var arch = new NeuralNetworkArchitecture<float>(inputType: InputType.FourDimensional,
    taskType: NeuralNetworkTaskType.Regression, inputFrames: 10, inputHeight: 64, inputWidth: 64, inputDepth: 3);
var model = new RVRT<float>(arch, new RVRTOptions());
model.Train(lowResClip, highResClip);         // [10, 3, 64, 64] -> [10, 3, 256, 256], pixel values 0-255
var upscaled = model.Upscale(lowResClip);

Constructors

Constructor Summary
RVRT(NeuralNetworkArchitecture<T> architecture, RVRTOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates an RVRT model in native training mode.
RVRT(NeuralNetworkArchitecture<T> architecture, string modelPath, RVRTOptions options) Creates an RVRT model in native training mode.

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape) Frames are pixels on the 0-255 scale; the network divides them by 255.
LayerInputDomain GetOutputDomain(int[] outputShape) Upscaled frames come back on the same 0-255 scale.
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> Upscale(Tensor<T> lowResFrames)