AMT<T>
Models & Types · AiDotNet.Video.FrameInterpolation
AMT: all-pairs multi-field transforms for efficient frame interpolation (Li et al., CVPR 2023).
Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → FrameInterpolationBase<T> → AMT<T>
For Beginners
AMT compares every pixel of one frame with every pixel of the other to find where things moved, predicts several candidate motions per pixel, and blends the frames moved along each into the in-between frame.
How It Works
The network follows the release (MCG-NKU/AMT, networks/AMT-S.py and AMT-L.py):
- Bidirectional all-pairs correlation. A RAFT feature encoder at 1/8 size; the correlation of every pixel
pair and its transpose, each pooled into a four-level pyramid.
- Coarse-to-fine flow decoding. An IFRNet context encoder and decoders from 1/16 size predict both flows
from the intermediate time and a feature; at 1/8, 1/4 and 1/2 size the flows, extended linearly to 0→1 and 1→0, look up both correlation pyramids and an update block refines flows and feature.
- Multi-field synthesis. The last decoder predicts several flow pairs, masks and residuals per pixel at
full size; each pair warps both frames, the blends are averaged and corrected by a small convolution.
Frames are padded by edge replication to multiples of 16 (InputPadder) and the mean of both frames is subtracted and restored. Any time in (0, 1) can be synthesized.
Training follows trainers/base_trainer.py and cfgs/AMT-S.yaml at the midpoint: AdamW at 2e-4 with cosine decay to 2e-5, on the Charbonnier and ternary (census) losses. The release’s third term, the multi-field flow loss, needs pseudo ground-truth flows from a pretrained flow network that a frame-pair → frame sample does not carry, so it is not applied.
Constructors
| Constructor | Summary |
|---|---|
AMT(NeuralNetworkArchitecture<T> architecture, AMTOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates an AMT model in native training mode. |
AMT(NeuralNetworkArchitecture<T> architecture, string modelPath, AMTOptions options) |
Creates an AMT model in native training mode. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
|
LayerInputDomain GetInputDomain(int[] inputShape) |
Frames are pixels on the 0-255 scale; the network divides them by 255. |
LayerInputDomain GetOutputDomain(int[] outputShape) |
The interpolated frame comes back on the same 0-255 scale. |
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Tensor<T> Interpolate(Tensor<T> frame0, Tensor<T> frame1, double t) |