TTVSR<T>

Models & Types · AiDotNet.Video.Enhancement

TTVSR: learning trajectory-aware transformer for long-range video super-resolution.

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoSuperResolutionBase<T> → TTVSR<T>

For Beginners

Imagine following a ball as it moves across frames. Instead of looking at the same fixed spot in every frame (which would miss the ball after it moves), TTVSR tracks where objects actually go and gathers information along their travel path, even from frames far back in the video.

How It Works

The network is the authors’ TTVSRNet (Liu et al., CVPR 2022; researchmm/TTVSR): SPyNet flow, per-frame features from five residual blocks, and backward and forward recurrent propagation through a shared 60-block trunk. At every step the propagated feature is warped by the flow and refined by trajectory-aware attention: each keyframe (every third frame) leaves 4x4 tokens at three scales and a trajectory, a location map warped along the flow at every later step, and the current frame takes, per token, the trajectory token of the most similar keyframe, scaled by that cosine similarity. The two directions are fused with the frame’s features and reconstructed at x4.

Training follows configs/TTVSR_reds4.py: Charbonnier loss, Adam (0.9, 0.99) at 2e-4 with SPyNet at 0.125 times the rate and frozen for the first 5,000 iterations, cosine annealing to 1e-7 over 400,000 iterations. SPyNet is loaded pretrained in the release and starts from PyTorch’s default initialization here.

Example

var arch = new NeuralNetworkArchitecture<float>(inputType: InputType.FourDimensional,
    taskType: NeuralNetworkTaskType.Regression, inputFrames: 10, inputHeight: 64, inputWidth: 64, inputDepth: 3);
var model = new TTVSR<float>(arch, new TTVSROptions());
model.Train(lowResClip, highResClip);         // [10, 3, 64, 64] -> [10, 3, 256, 256], pixel values 0-255
var upscaled = model.Upscale(lowResClip);

Constructors

Constructor Summary
TTVSR(NeuralNetworkArchitecture<T> architecture, TTVSROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a TTVSR model in native training mode.
TTVSR(NeuralNetworkArchitecture<T> architecture, string modelPath, TTVSROptions options) Creates a TTVSR model in native training mode.

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape) Frames are pixels on the 0-255 scale; the network divides them by 255.
LayerInputDomain GetOutputDomain(int[] outputShape) Upscaled frames come back on the same 0-255 scale.
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> Upscale(Tensor<T> lowResFrames)