SAM<T>
Models & Types · AiDotNet.ComputerVision.Segmentation.Foundation
Segment Anything Model (SAM): the first promptable foundation model for image segmentation.
Inheritance: NeuralNetworkBase<T> → SegmentationModelBase<T> → PromptableSegmentationBase<T> → SAM<T>
For Beginners
SAM can segment any object in any image given a prompt (point click, bounding box, or text). It was trained on the SA-1B dataset containing over 1 billion masks across 11 million images, making it extremely versatile.
Common use cases:
- Interactive object selection in image editors
- Automatic mask generation for datasets
- Zero-shot transfer to new domains without fine-tuning
- Foundation for downstream segmentation tasks
How It Works
Technical Details: the official network (facebookresearch/segment-anything), checked against the official code:
- ViT-B/L/H image encoder on 16x16 patches, windowed and global attention with relative positions, and a neck
- Prompt encoder for points, boxes and masks (random Fourier positions)
- Two-way transformer mask decoder with an IoU prediction head
- Ambiguity-aware: three masks for a single point
- SetImage encodes once (longest side resized to the encoder size); prompts then only run the decoder
- SegmentEverything is the official automatic mask generator; Train is the paper’s interactive training simulation
Reference: Kirillov et al., “Segment Anything”, ICCV 2023.
Example
// Create a SAM model for promptable image segmentation
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.ThreeDimensional,
taskType: NeuralNetworkTaskType.BinaryClassification,
inputHeight: 1024, inputWidth: 1024, inputDepth: 3, outputSize: 1);
var model = new SAM<double>(architecture);
// Every tunable is now set through the options object; these are the defaults:
var custom = new SAM<double>(architecture,
options: new SAMOptions { ModelSize = SAMModelSize.ViTHuge });
// Or load a pre-trained ONNX model for zero-shot segmentation
var onnxModel = new SAM<double>(architecture, "sam_vit_h.onnx");
Constructors
| Constructor | Summary |
|---|---|
SAM(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction, SAMOptions options) |
Creates a trainable SAM. |
SAM(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, SAMOptions options) |
Creates an inference-only SAM from an ONNX model. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
|
List<PromptedSegmentationResult<T>> SegmentEverything() |
The official automatic mask generator with its default settings: a 32×32 grid of foreground points, three masks per point, kept when the predicted IoU exceeds 0.88 and the stability score (offset 1.0) reaches 0.95, then box non-maximum supp… |
ModelMetadata<T> GetModelMetadata() |
|
PromptedSegmentationResult<T> SegmentFromBox(Tensor<T> box) |
The mask for a box [4] (x0, y0, x1, y1 in the image): a single unambiguous mask. |
PromptedSegmentationResult<T> SegmentFromMask(Tensor<T> mask) |
The mask for a mask prompt of logits: either low-resolution logits [4·g, 4·g] (a previous result’s LowResLogits plane, SamPredictor’s mask_input), or a map of the image’s size [H, W], which is brought to the prompt resolution the wa… |
PromptedSegmentationResult<T> SegmentFromPoints(Tensor<T> points, Tensor<T> labels) |
The three ambiguity-aware masks for points [k, 2] (x, y in the image) with labels [k] (1 foreground, 0 background), as SamPredictor gives them with multimask output. |
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
void SetImage(Tensor<T> image) |