ABME<T>
Models & Types · AiDotNet.Video.FrameInterpolation
ABME: asymmetric bilateral motion estimation for video frame interpolation (Park et al., ICCV 2021).
Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → FrameInterpolationBase<T> → ABME<T>
For Beginners
ABME first guesses one motion for both frames, mirrored around the missing middle frame, then lets the motion toward each frame differ, which handles speeding up, slowing down and curved paths, and finally blends the two frames moved to the middle into the missing frame.
How It Works
The pipeline follows the release (JunHeum/ABME, run.py):
- Symmetric bilateral motion (SBMENet, BMBC’s estimator with a strided first convolution). Both frames,
resized to multiples of 64 (halved from a height of 512), give the motion V from the intermediate time to the second frame at a quarter of that size; −V points to the first.
- Anchor and reliability. Both frames warped along ∓V average into the anchor frame, each filling the
other’s out-of-frame pixels; Z = exp(−20·|I₀→t − I₁→t|), the channel-averaged difference at a quarter size.
- Asymmetric bilateral motion refinement (ABMRNet). Run once from the anchor toward each frame, it frees
the two motions from the symmetric constraint.
- Synthesis (SynthesisNet). The frames and their feature pyramids, warped along the symmetric and the
asymmetric motions at three scales, are blended by per-pixel 3x3 dynamic filters over the four candidates and refined by a residual GridNet, at a multiple of 8, then resized back bicubically.
Training follows the paper’s three stages (Sec. 4.1), selected by Stage: the symmetric
estimator, then the refiner with the estimator fixed, each on the photometric loss of the frames warped to the
intermediate time (Charbonnier plus census); then all three networks end to end on the synthesized frame. Each
stage has its own Adam with the paper’s step halving.
The networks are trained for the midpoint only: Interpolate accepts t = 0.5.
Constructors
| Constructor | Summary |
|---|---|
ABME(NeuralNetworkArchitecture<T> architecture, ABMEOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates an ABME model in native training mode. |
ABME(NeuralNetworkArchitecture<T> architecture, string modelPath, ABMEOptions options) |
Creates an ABME model in native training mode. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
|
LayerInputDomain GetInputDomain(int[] inputShape) |
Frames are pixels on the 0-255 scale; the networks divide them by 255. |
LayerInputDomain GetOutputDomain(int[] outputShape) |
The interpolated frame comes back on the same 0-255 scale. |
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Tensor<T> Interpolate(Tensor<T> frame0, Tensor<T> frame1, double t) |