RAFT<T>

Models & Types · AiDotNet.Video.Motion

Recurrent All-pairs Field Transforms (RAFT) for optical flow estimation.

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → OpticalFlowBase<T> → RAFT<T>

For Beginners

RAFT is a state-of-the-art optical flow estimation model that predicts the motion between two consecutive video frames. Optical flow represents how pixels move from one frame to the next, useful for:

  • Motion analysis and tracking
  • Video stabilization
  • Action recognition
  • Video compression
  • Self-driving car perception

RAFT iteratively refines its flow estimate using a recurrent update mechanism, making it very accurate while remaining efficient.

How It Works

Technical Details:

  • Feature extraction using CNN encoder
  • 4D correlation volumes for all-pairs matching
  • GRU-based iterative update operator
  • Multi-scale feature pyramids

Reference: Teed and Deng, “RAFT: Recurrent All-Pairs Field Transforms for Optical Flow” ECCV 2020.

Example

// Create a RAFT model for recurrent optical flow estimation
var raft = new RAFT<double>();
            
// Or configure with custom architecture
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.ThreeDimensional,
    taskType: NeuralNetworkTaskType.Regression,
    inputHeight: 256, inputWidth: 256, inputDepth: 3, outputSize: 2);
var model = new RAFT<double>(architecture, options: new RAFTOptions { NumFeatures = 128, NumIterations = 12 });

Constructors

Constructor Summary
RAFT() Initializes a new instance with default architecture settings.
RAFT(NeuralNetworkArchitecture<T> architecture, RAFTOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Initializes a new instance of the RAFT class.

Properties

Property Summary
bool SupportsTraining { get; } Gets whether training is supported.

Methods

Method Summary
List<Tensor<T>> EstimateFlowIterative(Tensor<T> frame1, Tensor<T> frame2) Estimates optical flow with intermediate flow predictions.
ModelMetadata<T> GetModelMetadata()
Tensor<T> EstimateFlow(Tensor<T> frame1, Tensor<T> frame2) Estimates optical flow between two frames.
Tensor<T> ForwardForTraining(Tensor<T> input)