FlowFormer<T>

Models & Types · AiDotNet.Video.Motion

FlowFormer: A Transformer Architecture for Optical Flow.

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → OpticalFlowBase<T> → FlowFormer<T>

For Beginners

FlowFormer estimates optical flow - the apparent motion of objects between consecutive video frames. Unlike traditional methods, it uses transformers to capture long-range dependencies in the cost volume.

Optical flow is useful for:

  • Video stabilization
  • Object tracking
  • Action recognition
  • Video editing and effects

Example usage:

How It Works

Technical Details:

  • Transformer-based cost volume aggregation
  • Latent cost tokens for efficient memory
  • Iterative flow refinement
  • State-of-the-art accuracy on Sintel and KITTI benchmarks

Reference: “FlowFormer: A Transformer Architecture for Optical Flow” ECCV 2022 https://arxiv.org/abs/2203.16194

Constructors

Constructor Summary
FlowFormer()
FlowFormer(NeuralNetworkArchitecture<T> architecture, FlowFormerOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction)
FlowFormer(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, FlowFormerOptions options)

Properties

Property Summary
bool SupportsTraining { get; }

Methods

Method Summary
List<Tensor<T>> EstimateFlowForVideo(List<Tensor<T>> frames) Computes flow for all consecutive frame pairs in a video.
ModelMetadata<T> GetModelMetadata()
Tensor<T> EstimateFlow(Tensor<T> frame1, Tensor<T> frame2) Estimates optical flow between two frames.
Tensor<T> WarpWithFlow(Tensor<T> image, Tensor<T> flow) Warps an image using the estimated flow.
ValueTuple<Tensor<T>, Tensor<T>> EstimateBidirectionalFlow(Tensor<T> frame1, Tensor<T> frame2) Estimates bidirectional flow (forward and backward).