EfficientTAM<T>

Models & Types · AiDotNet.ComputerVision.Segmentation.Video

EfficientTAM (Efficient Track Anything): SAM 2’s promptable video tracking on a lightweight plain ViT, for devices.

Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → VideoSegmentationBase<T> → EfficientTAM<T>

For Beginners

Give EfficientTAM the masks of the objects to follow in the first frame, then feed it the next frames one at a time: it returns every object’s mask in each, remembering what the objects looked like.

How It Works

EfficientTAM (Xiong et al. 2024; the official yformer/EfficientTAM code) replaces SAM 2’s hierarchical Hiera encoder with a plain ViT-Tiny or ViT-Small (16×16 patches, 14×14 windows with four global blocks, a single-scale neck, and so no high-resolution skips in the mask decoder), and its “/2” models replace the memory cross-attention with an efficient one that pools each memory frame’s keys and values over 2×2 windows. Everything else is SAM 2’s network and tracker (SAM21, verified against the official code), and the streaming API tracks each object as the official video predictor does with its prompt on the first frame. Training follows the paper (see EfficientTAMOptions). Released checkpoints load with LoadPretrainedWeights (the official state dict saved as safetensors).

Reference: Xiong et al., “Efficient Track Anything”, arXiv 2024.

Constructors

Constructor Summary
EfficientTAM(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, EfficientTAMOptions options) Creates a trainable EfficientTAM.

Methods

Method Summary
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)
void AddCorrection(int objectId, Tensor<T> correctionMask) Corrects an object with its mask [H, W] on the next frame, which becomes one of its prompted frames.
void LoadPretrainedWeights(string path) Loads a checkpoint saved as safetensors: the official state dict (keys such as image_encoder.trunk.blocks.0.attn.qkv.weight) or its Hugging Face-style renaming.
void SetTrainingMode(bool isTraining)
void TrainOnVideo(Tensor<T> video, Tensor<T> masks) One step of the video training on one object: frames [F, 3, H, W] and their masks [F, H, W].