VFIMamba<T>

Models & Types · AiDotNet.Video.FrameInterpolation

VFIMamba state-space model for video frame interpolation with linear-complexity attention.

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → FrameInterpolationBase<T> → VFIMamba<T>

For Beginners

VFIMamba uses a new type of AI architecture called “Mamba” that can process long sequences efficiently (unlike transformers which get slow with long inputs). This means it can handle high-resolution frames without running out of memory, while still understanding how every pixel relates to every other pixel.

How It Works

References:

  • Paper: “VFIMamba: Video Frame Interpolation with State Space Models” (Zhang et al., NeurIPS 2024)

VFIMamba is the first to apply Mamba (selective state space models) to frame interpolation:

  • Selective state space: uses Mamba’s selective scan mechanism instead of attention, achieving

linear complexity O(N) for processing frame features while maintaining global context, enabling efficient processing of high-resolution frames

  • Bidirectional scanning: scans frame features in both forward (left-to-right, top-to-bottom)

and backward directions, ensuring each pixel has full global context from all directions

  • Cross-frame state propagation: the SSM state from one frame is propagated to condition

the processing of the other frame, enabling implicit motion correspondence without explicit flow estimation

  • Multi-scale Mamba blocks: hierarchical Mamba blocks at different spatial scales, capturing

both fine-grained texture details and coarse motion patterns

Example

// Create a VFIMamba model for state-space based frame interpolation
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.ThreeDimensional,
    taskType: NeuralNetworkTaskType.Generative,
    inputHeight: 256, inputWidth: 256, inputDepth: 3);
var options = new VFIMambaOptions();
var vfiMamba = new VFIMamba<double>(architecture, options);
            
// Or load a pre-trained ONNX model for inference
var vfiMambaOnnx = new VFIMamba<double>(architecture, "vfimamba_model.onnx");

Constructors

Constructor Summary
VFIMamba(NeuralNetworkArchitecture<T> architecture, VFIMambaOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a VFIMamba model for native training and inference.
VFIMamba(NeuralNetworkArchitecture<T> architecture, string modelPath, VFIMambaOptions options) Creates a VFIMamba model for native training and inference.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> Interpolate(Tensor<T> frame0, Tensor<T> frame1, double t)