SAM2<T>
Models & Types · AiDotNet.Video.Segmentation
SAM 2 (Segment Anything Model 2) for promptable segmentation of images and videos.
Inheritance: NeuralNetworkBase<T> → SAM2<T>
For Beginners
SAM 2 cuts an object out of an image, or follows it through a video, from a few clicks, a box or a mask. Give it clicks on the first frame and it segments the object in every frame, remembering what the object looked like in earlier frames so it can find it again after it moves or is hidden for a while.
How It Works
This is the network the paper describes and Meta released (facebookresearch/sam2): the Hiera image encoder with an FPN
neck, SAM’s prompt encoder and a mask decoder with an object-score head, a memory encoder that stores each frame’s
prediction, memory attention that conditions every new frame on the stored memories and the object pointers, and the
video predictor’s tracking (prompted frames decoded on their own, the object propagated forward and then backward).
It matches Hugging Face’s port of the official code at 1e-12, and loads the released checkpoints in Hugging Face’s
format with LoadPretrainedWeights. Training follows the paper’s recipe (Table 12b) and the official
SAM2Train: simulated mask, click and box prompts, corrective clicks, and the focal, dice, IoU and occlusion
losses (see SAM2Options).
Reference: Ravi et al., “SAM 2: Segment Anything in Images and Videos”, 2024.
Example
var architecture = new NeuralNetworkArchitecture<float>(
inputType: InputType.ThreeDimensional,
taskType: NeuralNetworkTaskType.BinaryClassification,
inputHeight: 1024, inputWidth: 1024, inputDepth: 3, outputSize: 1);
var sam2 = new SAM2<float>(architecture, options: new SAM2Options { ModelSize = SAM2ModelSize.Tiny });
sam2.LoadPretrainedWeights("sam2-hiera-tiny/model.safetensors");
var masks = sam2.TrackObject(frames, new float[,] { { 410, 220 } }, new[] { 1 });
Constructors
| Constructor | Summary |
|---|---|
SAM2() |
Creates SAM 2 (Base+) for 1024-pixel frames. |
SAM2(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, SAM2Options options) |
Creates SAM 2. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
|
LayerInputDomain GetOutputDomain(int[] outputShape) |
SAM 2’s masks are probabilities. |
List<Tensor<T>> InteractiveVideoSegmentation(List<Tensor<T>> frames, Dictionary<int, ValueTuple<float[], int[]>> framePrompts) |
Segments an object through a video from clicks on any frames: each prompted frame is decoded from its clicks, and the object is propagated from the first prompted frame forward to the end and back to the start. |
List<Tensor<T>> TrackObject(List<Tensor<T>> frames, float[] points, int[] pointLabels) |
Segments an object through a video from clicks on its first frame. |
List<Tensor<T>> TrackObjectWithBox(List<Tensor<T>> frames, float[] box) |
Segments an object through a video from a box [x0, y0, x1, y1] on its first frame. |
List<Tensor<T>> TrackObjectWithMask(List<Tensor<T>> frames, Tensor<T> mask) |
Segments an object through a video from its mask [H, W] on the first frame. |
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Tensor<T> SegmentWithBox(Tensor<T> image, float[] box) |
Segments an object in one image from a box [x0, y0, x1, y1]. |
Tensor<T> SegmentWithMask(Tensor<T> image, Tensor<T> lowResLogits) |
Refines a mask in one image from a previous prediction’s low-resolution logits [S/4, S/4] (S the encoder size), the image predictor’s mask_input. |
Tensor<T> SegmentWithPoints(Tensor<T> image, float[] points, int[] pointLabels) |
Segments an object in one image from clicks (the image predictor): with one click the best of the three ambiguity-aware masks by predicted IoU, otherwise the single mask. |
double GetOcclusionScore(Tensor<T> image, float[] points, int[] pointLabels) |
The probability that the clicked object is NOT visible in the image: one minus the sigmoid of the object-score head. |
void LoadPretrainedWeights(string path) |
Loads a released checkpoint in Hugging Face’s format (the model.safetensors of facebook/sam2-hiera-*, or facebook/sam2.1-hiera-* with Sam21); the model must be built with that checkpoint’s size. |
void SetTrainingMode(bool isTraining) |