GMFlow<T>
Models & Types · AiDotNet.Video.Motion
GMFlow (Global Matching Flow) for accurate optical flow estimation.
Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → OpticalFlowBase<T> → GMFlow<T>
For Beginners
GMFlow estimates how pixels move between video frames using a global matching approach. Unlike local methods that only look at small neighborhoods, GMFlow considers the entire image when matching pixels, making it better at:
- Large displacements (fast motion)
- Textureless regions
- Occlusions and disocclusions
- Repetitive patterns
The output is a “flow field” where each pixel has (dx, dy) values indicating where that pixel moved to in the next frame.
How It Works
Technical Details:
- Transformer-based global matching architecture
- Cross-attention for finding correspondences
- Hierarchical refinement for sub-pixel accuracy
- Self-attention for context aggregation
Reference: Xu et al., “GMFlow: Learning Optical Flow via Global Matching” CVPR 2022.
Example
// Create a GMFlow model for global matching optical flow estimation
var gmFlow = new GMFlow<double>();
// Or configure with custom architecture
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.ThreeDimensional,
taskType: NeuralNetworkTaskType.Regression,
inputHeight: 256, inputWidth: 256, inputDepth: 3, outputSize: 2);
var model = new GMFlow<double>(architecture, options: new GMFlowOptions { NumFeatures = 128 });
Constructors
| Constructor | Summary |
|---|---|
GMFlow() |
Initializes a new instance with default architecture settings. |
GMFlow(NeuralNetworkArchitecture<T> architecture, GMFlowOptions options) |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
Gets whether training is supported. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> EstimateFlow(Tensor<T> frame1, Tensor<T> frame2) |
Estimates optical flow between two frames. |
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Tensor<T> WarpImage(Tensor<T> image, Tensor<T> flow) |
Warps an image using the estimated flow. |
ValueTuple<Tensor<T>, Tensor<T>> EstimateBidirectionalFlow(Tensor<T> frame1, Tensor<T> frame2) |
Computes forward and backward flow for consistency checking. |
ValueTuple<Tensor<T>, Tensor<T>> EstimateFlowWithOcclusion(Tensor<T> frame1, Tensor<T> frame2) |
Estimates flow with occlusion mask. |