GroundedSAM2<T>

Models & Types · AiDotNet.VisionLanguage.Grounding

Grounded SAM 2: Grounding DINO finds the boxes a caption names, and SAM 2 segments each box.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → GroundedSAM2<T>

Implements: IGroundingTrainable<T>, IVisualGroundingModel<T>

For Beginners

Describe what you want (“cat . remote control .”) and Grounded SAM 2 returns a box and a pixel mask for each match.

How It Works

Grounded SAM (Ren et al., 2024) assembles open-world models rather than training a new one: an open-set detector turns text into boxes, and a promptable segmenter turns each box into a mask. Grounded SAM 2 (IDEA-Research/Grounded-SAM-2) pairs Grounding DINO with SAM 2. As the official demo runs them: Grounding DINO predicts with box and text thresholds 0.35 and 0.25; its boxes, scaled to pixels, prompt SAM 2’s image predictor (the image resized to 1024, single-mask output); and the masks are SAM 2’s logits resized to the image and thresholded at 0.

Both networks are the official ones (each checked against Hugging Face’s port). The paper trains neither; the detector’s own objective trains it here (TrainGrounding), and SAM 2 is used as loaded.

References:

  • Paper: “Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks” (Ren et al., 2024)

Example

var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.ThreeDimensional,
    taskType: NeuralNetworkTaskType.ImageSegmentation,
    inputHeight: 1024, inputWidth: 1024, inputDepth: 3, outputSize: 256);
var model = new GroundedSAM2<double>(architecture, new GroundedSAM2Options());
var segments = model.Segment(image, "cat . remote control .");

Constructors

Constructor Summary
GroundedSAM2(NeuralNetworkArchitecture<T> architecture, GroundedSAM2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)

Properties

Property Summary
int EmbeddingDimension { get; } The channels of SAM 2’s image embedding, which EncodeImage returns.
int MaxDetections { get; } The most detections GroundText returns (the highest scoring ones).

Methods

Method Summary
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
IReadOnlyList<GroundedSegment<T>> Segment(Tensor<T> image, string caption) The official demo: Grounding DINO’s detections of the caption’s phrases, each box (in pixels) prompting SAM 2 for a single mask, resized to the image and thresholded at MaskThreshold.
IReadOnlyList<GroundingDetection<T>> Ground(Tensor<T> image, string caption) Grounding DINO’s detections of the caption’s phrases (the official predict).
ModelMetadata<T> GetModelMetadata()
OpenVocabSegmentationResult<T> SegmentWithPrompt(Tensor<T> image, string prompt) Segments what one prompt describes: the union of every detection’s mask, as one class.
OpenVocabSegmentationResult<T> SegmentWithText(Tensor<T> image, IReadOnlyList<string> classNames) Segments the named classes: the class names joined into one caption (“a.
T EvaluateGroundingLoss(Tensor<T> image, string caption, IReadOnlyList<GroundingBox> boxes) The detector’s loss on an image, its caption and boxes, without training (dropout off).
Tensor<T> DetectObjects(Tensor<T> image, IReadOnlyList<string> categories) The official predict_with_classes: the categories joined as “a.
Tensor<T> EncodeImage(Tensor<T> image) SAM 2’s image embedding [SamDim, g, g] (or [B, SamDim, g, g]), g = SamImageSize / 16, of images in 0..1 resized to SamImageSize.
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> GroundText(Tensor<T> image, string textQuery) Ground as a tensor [n, 5] of (x1, y1, x2, y2, score) rows, the highest-scoring MaxDetections in query order.
void SetTrainingMode(bool isTraining)
void TrainGrounding(Tensor<T> image, string caption, IReadOnlyList<GroundingBox> boxes) One training step of the detector on an image, its caption and the boxes its phrases name, with Grounding DINO’s loss.