SAM<T>
Models & Types · AiDotNet.VisionLanguage.Encoders
Segment Anything Model (SAM) vision encoder for promptable image segmentation.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → SAM<T>
Implements: IVisualEncoder<T>
For Beginners
SAM (Segment Anything Model) from Meta is a promptable segmentation model — you give it an image and a prompt (a point click, a bounding box, or a rough mask) and it segments the object you indicated. Trained on over 1 billion masks, its ViT encoder uses windowed attention with occasional global attention for efficient high-resolution processing. Default values follow the original paper settings.
How It Works
SAM (Kirillov et al., 2023) consists of a ViT image encoder producing image embeddings, a prompt encoder that handles points/boxes/masks, and a lightweight mask decoder. The image encoder uses windowed attention with occasional global attention blocks for efficiency at high resolution.
References:
- Paper: “Segment Anything” (Kirillov et al., 2023)
Example
// Create a SAM model for promptable image segmentation
// using ViT encoder with windowed attention for high-resolution images
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 1024, inputWidth: 1024, inputDepth: 3, outputSize: 256);
// ONNX inference mode with pre-trained model
var model = new SAM<double>(architecture, "sam.onnx");
// Training mode with native layers
var trainModel = new SAM<double>(architecture, new SAMOptions());
Constructors
| Constructor | Summary |
|---|---|
SAM(NeuralNetworkArchitecture<T> architecture, SAMOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
SAM(NeuralNetworkArchitecture<T> architecture, string modelPath, SAMOptions options) |
Properties
| Property | Summary |
|---|---|
int EmbeddingDimension { get; } |
The image embedding’s channels (the mask decoder’s width, 256 in the released models). |
Methods
| Method | Summary |
|---|---|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> EncodeImage(Tensor<T> image) |
The image embedding [MaskDecoderDim, g, g] (or [B, MaskDecoderDim, g, g]) of images in 0..1 at most the encoder size: normalized by the options’ mean and std, zero-padded to a square, and run through SAM’s image encoder. |
Tensor<T> ForwardForTraining(Tensor<T> input) |