SoftSplat<T>

Models & Types · AiDotNet.Video.FrameInterpolation

SoftSplat: softmax splatting for video frame interpolation (Niklaus and Liu, CVPR 2020).

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → FrameInterpolationBase<T> → SoftSplat<T>

For Beginners

SoftSplat estimates motion both ways between the two frames, pushes each frame’s feature pyramid forward along its motion to the in-between time (blending pixels that land on the same spot by a learned importance), and then lets a grid of convolutions paint the in-between frame from the pushed features.

How It Works

The network follows the release (sniklaus/softmax-splatting, run.py and softsplat.py). Both frames are standardized by their joint mean and standard deviation. A PWC-Net (a six-level feature extractor and six decoders on 9x9 correlation volumes) estimates the flows in both directions. The synthesis extracts a three-level feature pyramid from each frame, computes each frame’s importance metric from its brightness-constancy error with a small U-Net, and softmax-splats every level, with its metric, forward to time t, the metric (scaled by 2t or 2(1 − t)) as the splatting weight. A three-stream GridNet fuses the splatted pyramids into the frame, and the statistics are restored. Sides are replicate-padded to even sizes, as run.estimate pads. Any time in [0, 1] can be synthesized.

Training follows the paper, which adopts the training of context-aware synthesis (Niklaus and Liu, CVPR 2018): AdaMax (β1 0.9, β2 0.999) at 1e-3 on the Laplacian loss Σ 2^(i−1)·|L_i(Î) − L_i(I)| over five pyramid levels at the midpoint, the flow estimator fine-tuned end to end. The norm is averaged over pixels rather than summed, which AdaMax’s scale-invariant step does not distinguish, and training crops are multiples of 16 so every pyramid level halves exactly. The release ships trained weights but no training script.

Constructors

Constructor Summary
SoftSplat(NeuralNetworkArchitecture<T> architecture, SoftSplatOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a SoftSplat model in native training mode.
SoftSplat(NeuralNetworkArchitecture<T> architecture, string modelPath, SoftSplatOptions options) Creates a SoftSplat model in native training mode.

Methods

Method Summary
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
LayerInputDomain GetInputDomain(int[] inputShape) Frames are pixels on the 0-255 scale; the network divides them by 255.
LayerInputDomain GetOutputDomain(int[] outputShape) The interpolated frame comes back on the same 0-255 scale.
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> Interpolate(Tensor<T> frame0, Tensor<T> frame1, double t)