SAM21<T>
Models & Types · AiDotNet.ComputerVision.Segmentation.Foundation
SAM 2.1: Segment Anything Model 2.1, Meta’s refreshed SAM 2 for images and videos.
Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → PromptableSegmentationBase<T> → SAM21<T>
For Beginners
SAM 2.1 cuts out an object from a few clicks, a box or a rough mask, in one image or through a
video. Call SetImage and then SegmentFromPoints (or box, mask) for an image, or
TrackObject for a video.
How It Works
SAM 2.1 is SAM 2’s network (Ravi et al. 2024; facebookresearch/sam2’s sam2.1_hiera_* configs) with the two
changes its release made: the object pointers carry a projected temporal encoding, and the memories of frames predicted
to be occluded get a learned embedding. It shares the network, tracking and training with
SAM2<T>, which match the official code at 1e-9: images go through the official image
predictor (resized to the encoder’s square, points and boxes scaled to it, masks resized back) and the automatic mask
generator; videos through the video predictor. Released checkpoints load with LoadPretrainedWeights.
Reference: Ravi et al., “SAM 2: Segment Anything in Images and Videos”, Meta AI, 2024.
Example
var architecture = new NeuralNetworkArchitecture<float>(
inputType: InputType.ThreeDimensional,
taskType: NeuralNetworkTaskType.BinaryClassification,
inputHeight: 1024, inputWidth: 1024, inputDepth: 3, outputSize: 1);
var sam = new SAM21<float>(architecture, options: new SAM21Options { ModelSize = SAM21ModelSize.Tiny });
sam.LoadPretrainedWeights("sam2.1-hiera-tiny/model.safetensors");
sam.SetImage(image);
var result = sam.SegmentFromPoints(points, labels);
Constructors
| Constructor | Summary |
|---|---|
SAM21(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, SAM21Options options) |
Creates a trainable SAM 2.1. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
|
List<PromptedSegmentationResult<T>> SegmentEverything() |
The official SAM2AutomaticMaskGenerator with its default settings: a 32×32 grid of foreground points, three masks per point, kept when the predicted IoU exceeds 0.8 and the stability score (offset 1.0) reaches 0.95, then box non-maximum sup… |
List<Tensor<T>> InteractiveVideoSegmentation(List<Tensor<T>> frames, Dictionary<int, ValueTuple<float[], int[]>> framePrompts) |
Segments an object through a video from clicks on any frames: each prompted frame is decoded from its clicks, and the object is propagated from the first prompted frame forward to the end and back to the start. |
List<Tensor<T>> TrackObject(List<Tensor<T>> frames, float[] points, int[] pointLabels) |
Segments an object through a video (frames [3, H, W], values in 0..1) from clicks [k, 2] on its first frame (labels 1 object, 0 background); each frame’s mask probabilities [H, W]. |
List<Tensor<T>> TrackObjectWithBox(List<Tensor<T>> frames, float[] box) |
Segments an object through a video from a box [x0, y0, x1, y1] on its first frame. |
List<Tensor<T>> TrackObjectWithMask(List<Tensor<T>> frames, Tensor<T> mask) |
Segments an object through a video from its mask [H, W] on the first frame. |
ModelMetadata<T> GetModelMetadata() |
|
PromptedSegmentationResult<T> SegmentFromBox(Tensor<T> box) |
The mask for a box [4] (x0, y0, x1, y1 in the image): a single mask. |
PromptedSegmentationResult<T> SegmentFromMask(Tensor<T> mask) |
The mask for a mask prompt of logits: low-resolution logits [S/4, S/4] (a previous result’s LowResLogits plane, the predictor’s mask_input), or a map of the image’s size [H, W], resized to that resolution (bilinear, antialiased). |
PromptedSegmentationResult<T> SegmentFromPoints(Tensor<T> points, Tensor<T> labels) |
The three ambiguity-aware masks for points [k, 2] (x, y in the image) with labels [k] (1 foreground, 0 background), as SAM2ImagePredictor gives them with multimask output. |
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
void LoadPretrainedWeights(string path) |
Loads a released checkpoint in Hugging Face’s format (the model.safetensors of facebook/sam2.1-hiera-*); the model must be built with that checkpoint’s size. |
void SetImage(Tensor<T> image) |
|
void SetTrainingMode(bool isTraining) |
|
void TrainOnVideo(Tensor<T> video, Tensor<T> masks) |
One step of SAM 2’s video training on one object: frames [F, 3, H, W] (values in 0..1) and their masks [F, H, W]. |