EfficientTAM<T>
Models & Types · AiDotNet.ComputerVision.Segmentation.Video
EfficientTAM (Efficient Track Anything): SAM 2’s promptable video tracking on a lightweight plain ViT, for devices.
Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → VideoSegmentationBase<T> → EfficientTAM<T>
For Beginners
Give EfficientTAM the masks of the objects to follow in the first frame, then feed it the next frames one at a time: it returns every object’s mask in each, remembering what the objects looked like.
How It Works
EfficientTAM (Xiong et al. 2024; the official yformer/EfficientTAM code) replaces SAM 2’s hierarchical Hiera encoder with
a plain ViT-Tiny or ViT-Small (16×16 patches, 14×14 windows with four global blocks, a single-scale neck, and so no
high-resolution skips in the mask decoder), and its “/2” models replace the memory cross-attention with an efficient
one that pools each memory frame’s keys and values over 2×2 windows. Everything else is SAM 2’s network and tracker
(SAM21, verified against the official code), and the streaming API tracks each object as
the official video predictor does with its prompt on the first frame. Training follows the paper (see
EfficientTAMOptions). Released checkpoints load with LoadPretrainedWeights (the official
state dict saved as safetensors).
Reference: Xiong et al., “Efficient Track Anything”, arXiv 2024.
Constructors
| Constructor | Summary |
|---|---|
EfficientTAM(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, EfficientTAMOptions options) |
Creates a trainable EfficientTAM. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
void AddCorrection(int objectId, Tensor<T> correctionMask) |
Corrects an object with its mask [H, W] on the next frame, which becomes one of its prompted frames. |
void LoadPretrainedWeights(string path) |
Loads a checkpoint saved as safetensors: the official state dict (keys such as image_encoder.trunk.blocks.0.attn.qkv.weight) or its Hugging Face-style renaming. |
void SetTrainingMode(bool isTraining) |
|
void TrainOnVideo(Tensor<T> video, Tensor<T> masks) |
One step of the video training on one object: frames [F, 3, H, W] and their masks [F, H, W]. |