Cutie<T>
Models & Types · AiDotNet.Video.Segmentation
Cutie: Cutting-edge Video Instance Segmentation with transformer memory.
Inheritance: NeuralNetworkBase<T> → Cutie<T>
For Beginners
Cutie tracks objects in video. You provide a mask showing where the object is in the first frame, and Cutie finds that same object in all following frames - even when it moves, changes shape, or gets partially hidden.
Key features:
- Object permanence understanding (tracks objects even when briefly occluded)
- Efficient memory management for long videos
- High-quality mask predictions
- Multi-object tracking support
Example usage (native mode for training):
Example usage (ONNX mode for inference):
How It Works
Cutie is designed for semi-supervised video object segmentation (VOS). Given a mask for an object in the first frame, Cutie tracks and segments that object throughout the entire video with high accuracy.
Reference: “Putting the Object Back into Video Object Segmentation”
https://arxiv.org/abs/2310.12982
Constructors
| Constructor | Summary |
|---|---|
Cutie() |
Creates a Cutie model using native layers for training and inference. |
Cutie(NeuralNetworkArchitecture<T> architecture, CutieOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
|
Cutie(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, CutieOptions options) |
Creates a Cutie model using a pretrained ONNX model for inference. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
Gets whether training is supported (only in native mode). |
Methods
| Method | Summary |
|---|---|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
|
List<Tensor<T>> TrackObject(List<Tensor<T>> frames, Tensor<T> initialMask) |
Tracks and segments an object across video frames. |
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
Training forward. |
Tensor<T> SegmentFrame(Tensor<T> frame) |
Segments a single frame using the current memory state. |
void AddObject(Tensor<T> frame, Tensor<T> mask) |
Adds a new object to track by storing its features in memory. |
void ClearMemory() |
Clears the memory bank. |