SAM<T>

Models & Types · AiDotNet.ComputerVision.Segmentation.Foundation

Segment Anything Model (SAM): the first promptable foundation model for image segmentation.

Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → PromptableSegmentationBase<T> → SAM<T>

For Beginners

SAM can segment any object in any image given a prompt (point click, bounding box, or text). It was trained on the SA-1B dataset containing over 1 billion masks across 11 million images, making it extremely versatile.

Common use cases:

  • Interactive object selection in image editors
  • Automatic mask generation for datasets
  • Zero-shot transfer to new domains without fine-tuning
  • Foundation for downstream segmentation tasks

How It Works

Technical Details: the official network (facebookresearch/segment-anything), checked against the official code:

  • ViT-B/L/H image encoder on 16x16 patches, windowed and global attention with relative positions, and a neck
  • Prompt encoder for points, boxes and masks (random Fourier positions)
  • Two-way transformer mask decoder with an IoU prediction head
  • Ambiguity-aware: three masks for a single point
  • SetImage encodes once (longest side resized to the encoder size); prompts then only run the decoder
  • SegmentEverything is the official automatic mask generator; Train is the paper’s interactive training simulation

Reference: Kirillov et al., “Segment Anything”, ICCV 2023.

Example

// Create a SAM model for promptable image segmentation
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.ThreeDimensional,
    taskType: NeuralNetworkTaskType.BinaryClassification,
    inputHeight: 1024, inputWidth: 1024, inputDepth: 3, outputSize: 1);
var model = new SAM<double>(architecture);
// Every tunable is now set through the options object; these are the defaults:
var custom = new SAM<double>(architecture,
    options: new SAMOptions { ModelSize = SAMModelSize.ViTHuge });
            
// Or load a pre-trained ONNX model for zero-shot segmentation
var onnxModel = new SAM<double>(architecture, "sam_vit_h.onnx");

Constructors

Constructor Summary
SAM(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, SAMOptions options) Creates a trainable SAM.
SAM(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, SAMOptions options) Creates an inference-only SAM from an ONNX model.

Methods

Method Summary
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
List<PromptedSegmentationResult<T>> SegmentEverything() The official automatic mask generator with its default settings: a 32×32 grid of foreground points, three masks per point, kept when the predicted IoU exceeds 0.88 and the stability score (offset 1.0) reaches 0.95, then box non-maximum supp…
ModelMetadata<T> GetModelMetadata()
PromptedSegmentationResult<T> SegmentFromBox(Tensor<T> box) The mask for a box [4] (x0, y0, x1, y1 in the image): a single unambiguous mask.
PromptedSegmentationResult<T> SegmentFromMask(Tensor<T> mask) The mask for a mask prompt of logits: either low-resolution logits [4·g, 4·g] (a previous result’s LowResLogits plane, SamPredictor’s mask_input), or a map of the image’s size [H, W], which is brought to the prompt resolution the wa…
PromptedSegmentationResult<T> SegmentFromPoints(Tensor<T> points, Tensor<T> labels) The three ambiguity-aware masks for points [k, 2] (x, y in the image) with labels [k] (1 foreground, 0 background), as SamPredictor gives them with multimask output.
Tensor<T> ForwardForTraining(Tensor<T> input)
void SetImage(Tensor<T> image)