MIAVSR<T>

Models & Types · AiDotNet.Video.Enhancement

MIA-VSR: masked inter and intra-frame attention for efficient video super-resolution.

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoSuperResolutionBase<T> → MIAVSR<T>

For Beginners

MIA-VSR makes video super-resolution faster by being selective. Instead of comparing every pixel with every other pixel in neighboring frames (which is very slow), it uses “masks” to focus only on the most important parts. It has two types: inter-frame masks find the best matching regions across time, and intra-frame masks enhance spatial details within each frame.

Usage:

How It Works

MIA-VSR (Zhou et al., CVPR 2024) is a recurrent video super-resolution transformer:

  • Bidirectional second-order propagation (BasicVSR++’s grid): four branches, backward, forward,

backward, forward, each refining every frame from the two frames it already enhanced

  • Inter-and-intra-frame attention: queries from the current frame attend, inside 8x8 windows,

to keys and values from the current frame and the two previous enhanced frames, aligned by SPyNet patch alignment (PSRT)

  • Adaptive masked processing: each block predicts which positions changed since the previous

frame and reuses last frame’s result for the rest, trained with a Gumbel-softmax mask and a sparsity loss, and run sparsely at inference

  • Pixel-shuffle reconstruction plus a bilinear residual

Reference: “MIA-VSR: Masked Inter and Intra-Frame Attention for Video Super-Resolution” (Zhou et al., CVPR 2024)

Constructors

Constructor Summary
MIAVSR(NeuralNetworkArchitecture<T> architecture, MIAVSROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, SpyNetLayer<T> flowEstimator) Creates a MIA-VSR model in native training mode.
MIAVSR(NeuralNetworkArchitecture<T> architecture, string modelPath, MIAVSROptions options) Creates a MIA-VSR model in ONNX inference mode.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> Upscale(Tensor<T> lowResFrames)
void TrainWithGradientAccumulation(Tensor<T> input, Tensor<T> target, int batchSize)