SoftSplat<T>
Models & Types · AiDotNet.Video.FrameInterpolation
SoftSplat: softmax splatting for video frame interpolation (Niklaus and Liu, CVPR 2020).
Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → FrameInterpolationBase<T> → SoftSplat<T>
For Beginners
SoftSplat estimates motion both ways between the two frames, pushes each frame’s feature pyramid forward along its motion to the in-between time (blending pixels that land on the same spot by a learned importance), and then lets a grid of convolutions paint the in-between frame from the pushed features.
How It Works
The network follows the release (sniklaus/softmax-splatting, run.py and softsplat.py). Both frames are standardized by their joint mean and standard deviation. A PWC-Net (a six-level feature extractor and six decoders on 9x9 correlation volumes) estimates the flows in both directions. The synthesis extracts a three-level feature pyramid from each frame, computes each frame’s importance metric from its brightness-constancy error with a small U-Net, and softmax-splats every level, with its metric, forward to time t, the metric (scaled by 2t or 2(1 − t)) as the splatting weight. A three-stream GridNet fuses the splatted pyramids into the frame, and the statistics are restored. Sides are replicate-padded to even sizes, as run.estimate pads. Any time in [0, 1] can be synthesized.
Training follows the paper, which adopts the training of context-aware synthesis (Niklaus and Liu, CVPR 2018): AdaMax (β1 0.9, β2 0.999) at 1e-3 on the Laplacian loss Σ 2^(i−1)·|L_i(Î) − L_i(I)| over five pyramid levels at the midpoint, the flow estimator fine-tuned end to end. The norm is averaged over pixels rather than summed, which AdaMax’s scale-invariant step does not distinguish, and training crops are multiples of 16 so every pyramid level halves exactly. The release ships trained weights but no training script.
Constructors
| Constructor | Summary |
|---|---|
SoftSplat(NeuralNetworkArchitecture<T> architecture, SoftSplatOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a SoftSplat model in native training mode. |
SoftSplat(NeuralNetworkArchitecture<T> architecture, string modelPath, SoftSplatOptions options) |
Creates a SoftSplat model in native training mode. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
|
LayerInputDomain GetInputDomain(int[] inputShape) |
Frames are pixels on the 0-255 scale; the network divides them by 255. |
LayerInputDomain GetOutputDomain(int[] outputShape) |
The interpolated frame comes back on the same 0-255 scale. |
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Tensor<T> Interpolate(Tensor<T> frame0, Tensor<T> frame1, double t) |