GroundingDINO<T>

Models & Types · AiDotNet.VisionLanguage.Grounding

Grounding DINO: open-set detection combining DINO with grounded pre-training.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → GroundingDINO<T>

Implements: IGroundingTrainable<T>, IVisualGroundingModel<T>

For Beginners

Grounding DINO detects objects that you describe in words, so it is not limited to a fixed list of categories. Default values follow the released model.

How It Works

Grounding DINO (Liu et al. 2023) finds the regions of an image that a caption’s phrases name. A Swin image backbone and a BERT text backbone feed a feature enhancer that fuses the two (vision-text bi-attention, text self-attention, deformable image self-attention); language-guided query selection keeps the 900 encoder tokens most similar to the text; a cross-modality decoder refines them into boxes; and each box is scored against every caption token. The network is the official one (checked against Hugging Face’s port of IDEA-Research/GroundingDINO to 1e-8).

A caption lists what to find as period-separated phrases (“cat . remote control .”). Ground returns every query whose best token probability exceeds the box threshold (0.35), with the words whose tokens exceed the text threshold (0.25), as the official predict does. Images are read at the size given, in 0..1, and normalized with the ImageNet mean and deviation; the official demo resizes the shorter side to 800 first.

Training takes grounded phrases (TrainGrounding): the paper’s loss of Hungarian matching, a contrastive focal loss over the caption’s tokens, and L1 and GIoU box losses, on every decoder layer and the encoder’s selected queries. A tensor target cannot carry phrases, so Train refuses one.

References:

  • Paper: “Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection” (Liu et al., 2023)

Example

var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.ThreeDimensional,
    taskType: NeuralNetworkTaskType.ObjectDetection,
    inputHeight: 800, inputWidth: 800, inputDepth: 3, outputSize: 256);
var model = new GroundingDINO<double>(architecture, new GroundingDINOOptions());
var detections = model.Ground(image, "cat . remote control .");

Constructors

Constructor Summary
GroundingDINO(NeuralNetworkArchitecture<T> architecture, GroundingDINOOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)

Properties

Property Summary
int EmbeddingDimension { get; } The width of the image tokens EncodeImage returns.
int MaxDetections { get; } The most detections GroundText returns (the highest scoring ones).

Methods

Method Summary
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
IReadOnlyList<GroundingDetection<T>> Ground(Tensor<T> image, string caption) The regions the caption’s phrases name (the official predict): each query whose best token probability exceeds the box threshold (ConfidenceThreshold), with that probability, its box as fractions of the image, and the words whose tokens…
ModelMetadata<T> GetModelMetadata()
T EvaluateGroundingLoss(Tensor<T> image, string caption, IReadOnlyList<GroundingBox> boxes) The paper’s loss on an image, its caption and boxes, without training (dropout off).
Tensor<T> DetectObjects(Tensor<T> image, IReadOnlyList<string> categories) The official predict_with_classes: the categories joined as “a.
Tensor<T> EncodeImage(Tensor<T> image) The image tokens before they meet the text: every feature level (the backbone’s and the extra stride-2 level), projected to the model width and group-normalized, flattened and concatenated, [tokens, DecoderDim] (or `[B, tokens, DecoderDim…
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> GroundText(Tensor<T> image, string textQuery) Ground as a tensor [n, 5] of (x1, y1, x2, y2, score) rows, the highest-scoring MaxDetections in query order.
void SetTrainingMode(bool isTraining)
void TrainGrounding(Tensor<T> image, string caption, IReadOnlyList<GroundingBox> boxes) One training step on an image, its caption and the boxes its phrases name, with the paper’s loss (Hungarian matching; contrastive focal, L1 and GIoU losses on every decoder layer and the encoder’s selected queries).