SAM21<T>

Models & Types · AiDotNet.ComputerVision.Segmentation.Foundation

SAM 2.1: Segment Anything Model 2.1, Meta’s refreshed SAM 2 for images and videos.

Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → PromptableSegmentationBase<T> → SAM21<T>

For Beginners

SAM 2.1 cuts out an object from a few clicks, a box or a rough mask, in one image or through a video. Call SetImage and then SegmentFromPoints (or box, mask) for an image, or TrackObject for a video.

How It Works

SAM 2.1 is SAM 2’s network (Ravi et al. 2024; facebookresearch/sam2’s sam2.1_hiera_* configs) with the two changes its release made: the object pointers carry a projected temporal encoding, and the memories of frames predicted to be occluded get a learned embedding. It shares the network, tracking and training with SAM2<T>, which match the official code at 1e-9: images go through the official image predictor (resized to the encoder’s square, points and boxes scaled to it, masks resized back) and the automatic mask generator; videos through the video predictor. Released checkpoints load with LoadPretrainedWeights.

Reference: Ravi et al., “SAM 2: Segment Anything in Images and Videos”, Meta AI, 2024.

Example

var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.ThreeDimensional,
    taskType: NeuralNetworkTaskType.BinaryClassification,
    inputHeight: 1024, inputWidth: 1024, inputDepth: 3, outputSize: 1);
var sam = new SAM21<float>(architecture, options: new SAM21Options { ModelSize = SAM21ModelSize.Tiny });
sam.LoadPretrainedWeights("sam2.1-hiera-tiny/model.safetensors");
sam.SetImage(image);
var result = sam.SegmentFromPoints(points, labels);

Constructors

Constructor Summary
SAM21(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, SAM21Options options) Creates a trainable SAM 2.1.

Methods

Method Summary
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
List<PromptedSegmentationResult<T>> SegmentEverything() The official SAM2AutomaticMaskGenerator with its default settings: a 32×32 grid of foreground points, three masks per point, kept when the predicted IoU exceeds 0.8 and the stability score (offset 1.0) reaches 0.95, then box non-maximum sup…
List<Tensor<T>> InteractiveVideoSegmentation(List<Tensor<T>> frames, Dictionary<int, ValueTuple<float[], int[]>> framePrompts) Segments an object through a video from clicks on any frames: each prompted frame is decoded from its clicks, and the object is propagated from the first prompted frame forward to the end and back to the start.
List<Tensor<T>> TrackObject(List<Tensor<T>> frames, float[] points, int[] pointLabels) Segments an object through a video (frames [3, H, W], values in 0..1) from clicks [k, 2] on its first frame (labels 1 object, 0 background); each frame’s mask probabilities [H, W].
List<Tensor<T>> TrackObjectWithBox(List<Tensor<T>> frames, float[] box) Segments an object through a video from a box [x0, y0, x1, y1] on its first frame.
List<Tensor<T>> TrackObjectWithMask(List<Tensor<T>> frames, Tensor<T> mask) Segments an object through a video from its mask [H, W] on the first frame.
ModelMetadata<T> GetModelMetadata()
PromptedSegmentationResult<T> SegmentFromBox(Tensor<T> box) The mask for a box [4] (x0, y0, x1, y1 in the image): a single mask.
PromptedSegmentationResult<T> SegmentFromMask(Tensor<T> mask) The mask for a mask prompt of logits: low-resolution logits [S/4, S/4] (a previous result’s LowResLogits plane, the predictor’s mask_input), or a map of the image’s size [H, W], resized to that resolution (bilinear, antialiased).
PromptedSegmentationResult<T> SegmentFromPoints(Tensor<T> points, Tensor<T> labels) The three ambiguity-aware masks for points [k, 2] (x, y in the image) with labels [k] (1 foreground, 0 background), as SAM2ImagePredictor gives them with multimask output.
Tensor<T> ForwardForTraining(Tensor<T> input)
void LoadPretrainedWeights(string path) Loads a released checkpoint in Hugging Face’s format (the model.safetensors of facebook/sam2.1-hiera-*); the model must be built with that checkpoint’s size.
void SetImage(Tensor<T> image)
void SetTrainingMode(bool isTraining)
void TrainOnVideo(Tensor<T> video, Tensor<T> masks) One step of SAM 2’s video training on one object: frames [F, 3, H, W] (values in 0..1) and their masks [F, H, W].