GMFlow<T>

Models & Types · AiDotNet.Video.Motion

GMFlow (Global Matching Flow) for accurate optical flow estimation.

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → OpticalFlowBase<T> → GMFlow<T>

For Beginners

GMFlow estimates how pixels move between video frames using a global matching approach. Unlike local methods that only look at small neighborhoods, GMFlow considers the entire image when matching pixels, making it better at:

  • Large displacements (fast motion)
  • Textureless regions
  • Occlusions and disocclusions
  • Repetitive patterns

The output is a “flow field” where each pixel has (dx, dy) values indicating where that pixel moved to in the next frame.

How It Works

Technical Details:

  • Transformer-based global matching architecture
  • Cross-attention for finding correspondences
  • Hierarchical refinement for sub-pixel accuracy
  • Self-attention for context aggregation

Reference: Xu et al., “GMFlow: Learning Optical Flow via Global Matching” CVPR 2022.

Example

// Create a GMFlow model for global matching optical flow estimation
var gmFlow = new GMFlow<double>();
            
// Or configure with custom architecture
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.ThreeDimensional,
    taskType: NeuralNetworkTaskType.Regression,
    inputHeight: 256, inputWidth: 256, inputDepth: 3, outputSize: 2);
var model = new GMFlow<double>(architecture, options: new GMFlowOptions { NumFeatures = 128 });

Constructors

Constructor Summary
GMFlow() Initializes a new instance with default architecture settings.
GMFlow(NeuralNetworkArchitecture<T> architecture, GMFlowOptions options)

Properties

Property Summary
bool SupportsTraining { get; } Gets whether training is supported.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> EstimateFlow(Tensor<T> frame1, Tensor<T> frame2) Estimates optical flow between two frames.
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> WarpImage(Tensor<T> image, Tensor<T> flow) Warps an image using the estimated flow.
ValueTuple<Tensor<T>, Tensor<T>> EstimateBidirectionalFlow(Tensor<T> frame1, Tensor<T> frame2) Computes forward and backward flow for consistency checking.
ValueTuple<Tensor<T>, Tensor<T>> EstimateFlowWithOcclusion(Tensor<T> frame1, Tensor<T> frame2) Estimates flow with occlusion mask.