MIAVSR<T>
Models & Types · AiDotNet.Video.Enhancement
MIA-VSR: masked inter and intra-frame attention for efficient video super-resolution.
Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoSuperResolutionBase<T> → MIAVSR<T>
For Beginners
MIA-VSR makes video super-resolution faster by being selective. Instead of comparing every pixel with every other pixel in neighboring frames (which is very slow), it uses “masks” to focus only on the most important parts. It has two types: inter-frame masks find the best matching regions across time, and intra-frame masks enhance spatial details within each frame.
Usage:
How It Works
MIA-VSR (Zhou et al., CVPR 2024) is a recurrent video super-resolution transformer:
- Bidirectional second-order propagation (BasicVSR++’s grid): four branches, backward, forward,
backward, forward, each refining every frame from the two frames it already enhanced
- Inter-and-intra-frame attention: queries from the current frame attend, inside 8x8 windows,
to keys and values from the current frame and the two previous enhanced frames, aligned by SPyNet patch alignment (PSRT)
- Adaptive masked processing: each block predicts which positions changed since the previous
frame and reuses last frame’s result for the rest, trained with a Gumbel-softmax mask and a sparsity loss, and run sparsely at inference
- Pixel-shuffle reconstruction plus a bilinear residual
Reference: “MIA-VSR: Masked Inter and Intra-Frame Attention for Video Super-Resolution” (Zhou et al., CVPR 2024)
Constructors
| Constructor | Summary |
|---|---|
MIAVSR(NeuralNetworkArchitecture<T> architecture, MIAVSROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, SpyNetLayer<T> flowEstimator) |
Creates a MIA-VSR model in native training mode. |
MIAVSR(NeuralNetworkArchitecture<T> architecture, string modelPath, MIAVSROptions options) |
Creates a MIA-VSR model in ONNX inference mode. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Tensor<T> Upscale(Tensor<T> lowResFrames) |
|
void TrainWithGradientAccumulation(Tensor<T> input, Tensor<T> target, int batchSize) |