FlowFormer<T>
Models & Types · AiDotNet.Video.Motion
FlowFormer: A Transformer Architecture for Optical Flow.
Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → OpticalFlowBase<T> → FlowFormer<T>
For Beginners
FlowFormer estimates optical flow - the apparent motion of objects between consecutive video frames. Unlike traditional methods, it uses transformers to capture long-range dependencies in the cost volume.
Optical flow is useful for:
- Video stabilization
- Object tracking
- Action recognition
- Video editing and effects
Example usage:
How It Works
Technical Details:
- Transformer-based cost volume aggregation
- Latent cost tokens for efficient memory
- Iterative flow refinement
- State-of-the-art accuracy on Sintel and KITTI benchmarks
Reference: “FlowFormer: A Transformer Architecture for Optical Flow” ECCV 2022
https://arxiv.org/abs/2203.16194
Constructors
| Constructor | Summary |
|---|---|
FlowFormer() |
|
FlowFormer(NeuralNetworkArchitecture<T> architecture, FlowFormerOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
|
FlowFormer(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, FlowFormerOptions options) |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
Methods
| Method | Summary |
|---|---|
List<Tensor<T>> EstimateFlowForVideo(List<Tensor<T>> frames) |
Computes flow for all consecutive frame pairs in a video. |
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> EstimateFlow(Tensor<T> frame1, Tensor<T> frame2) |
Estimates optical flow between two frames. |
Tensor<T> WarpWithFlow(Tensor<T> image, Tensor<T> flow) |
Warps an image using the estimated flow. |
ValueTuple<Tensor<T>, Tensor<T>> EstimateBidirectionalFlow(Tensor<T> frame1, Tensor<T> frame2) |
Estimates bidirectional flow (forward and backward). |