Cutie<T>

Models & Types · AiDotNet.Video.Segmentation

Cutie: Cutting-edge Video Instance Segmentation with transformer memory.

Inheritance: NeuralNetworkBase<T> → Cutie<T>

For Beginners

Cutie tracks objects in video. You provide a mask showing where the object is in the first frame, and Cutie finds that same object in all following frames - even when it moves, changes shape, or gets partially hidden.

Key features:

  • Object permanence understanding (tracks objects even when briefly occluded)
  • Efficient memory management for long videos
  • High-quality mask predictions
  • Multi-object tracking support

Example usage (native mode for training):

Example usage (ONNX mode for inference):

How It Works

Cutie is designed for semi-supervised video object segmentation (VOS). Given a mask for an object in the first frame, Cutie tracks and segments that object throughout the entire video with high accuracy.

Reference: “Putting the Object Back into Video Object Segmentation” https://arxiv.org/abs/2310.12982

Constructors

Constructor Summary
Cutie() Creates a Cutie model using native layers for training and inference.
Cutie(NeuralNetworkArchitecture<T> architecture, CutieOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction)
Cutie(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, CutieOptions options) Creates a Cutie model using a pretrained ONNX model for inference.

Properties

Property Summary
bool SupportsTraining { get; } Gets whether training is supported (only in native mode).

Methods

Method Summary
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
List<Tensor<T>> TrackObject(List<Tensor<T>> frames, Tensor<T> initialMask) Tracks and segments an object across video frames.
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input) Training forward.
Tensor<T> SegmentFrame(Tensor<T> frame) Segments a single frame using the current memory state.
void AddObject(Tensor<T> frame, Tensor<T> mask) Adds a new object to track by storing its features in memory.
void ClearMemory() Clears the memory bank.