SAM<T>

Models & Types · AiDotNet.VisionLanguage.Encoders

Segment Anything Model (SAM) vision encoder for promptable image segmentation.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → SAM<T>

Implements: IVisualEncoder<T>

For Beginners

SAM (Segment Anything Model) from Meta is a promptable segmentation model — you give it an image and a prompt (a point click, a bounding box, or a rough mask) and it segments the object you indicated. Trained on over 1 billion masks, its ViT encoder uses windowed attention with occasional global attention for efficient high-resolution processing. Default values follow the original paper settings.

How It Works

SAM (Kirillov et al., 2023) consists of a ViT image encoder producing image embeddings, a prompt encoder that handles points/boxes/masks, and a lightweight mask decoder. The image encoder uses windowed attention with occasional global attention blocks for efficiency at high resolution.

References:

  • Paper: “Segment Anything” (Kirillov et al., 2023)

Example

// Create a SAM model for promptable image segmentation
// using ViT encoder with windowed attention for high-resolution images
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 1024, inputWidth: 1024, inputDepth: 3, outputSize: 256);
            
// ONNX inference mode with pre-trained model
var model = new SAM<double>(architecture, "sam.onnx");
            
// Training mode with native layers
var trainModel = new SAM<double>(architecture, new SAMOptions());

Constructors

Constructor Summary
SAM(NeuralNetworkArchitecture<T> architecture, SAMOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
SAM(NeuralNetworkArchitecture<T> architecture, string modelPath, SAMOptions options)

Properties

Property Summary
int EmbeddingDimension { get; } The image embedding’s channels (the mask decoder’s width, 256 in the released models).

Methods

Method Summary
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
ModelMetadata<T> GetModelMetadata()
Tensor<T> EncodeImage(Tensor<T> image) The image embedding [MaskDecoderDim, g, g] (or [B, MaskDecoderDim, g, g]) of images in 0..1 at most the encoder size: normalized by the options’ mean and std, zero-padded to a square, and run through SAM’s image encoder.
Tensor<T> ForwardForTraining(Tensor<T> input)