GroundingDINO<T>
Models & Types · AiDotNet.VisionLanguage.Grounding
Grounding DINO: open-set detection combining DINO with grounded pre-training.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → GroundingDINO<T>
Implements: IGroundingTrainable<T>, IVisualGroundingModel<T>
For Beginners
Grounding DINO detects objects that you describe in words, so it is not limited to a fixed list of categories. Default values follow the released model.
How It Works
Grounding DINO (Liu et al. 2023) finds the regions of an image that a caption’s phrases name. A Swin image backbone and a BERT text backbone feed a feature enhancer that fuses the two (vision-text bi-attention, text self-attention, deformable image self-attention); language-guided query selection keeps the 900 encoder tokens most similar to the text; a cross-modality decoder refines them into boxes; and each box is scored against every caption token. The network is the official one (checked against Hugging Face’s port of IDEA-Research/GroundingDINO to 1e-8).
A caption lists what to find as period-separated phrases (“cat . remote control .”). Ground returns
every query whose best token probability exceeds the box threshold (0.35), with the words whose tokens exceed the
text threshold (0.25), as the official predict does. Images are read at the size given, in 0..1, and
normalized with the ImageNet mean and deviation; the official demo resizes the shorter side to 800 first.
Training takes grounded phrases (TrainGrounding): the paper’s loss of Hungarian matching, a
contrastive focal loss over the caption’s tokens, and L1 and GIoU box losses, on every decoder layer and the
encoder’s selected queries. A tensor target cannot carry phrases, so Train refuses one.
References:
- Paper: “Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection” (Liu et al., 2023)
Example
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.ThreeDimensional,
taskType: NeuralNetworkTaskType.ObjectDetection,
inputHeight: 800, inputWidth: 800, inputDepth: 3, outputSize: 256);
var model = new GroundingDINO<double>(architecture, new GroundingDINOOptions());
var detections = model.Ground(image, "cat . remote control .");
Constructors
| Constructor | Summary |
|---|---|
GroundingDINO(NeuralNetworkArchitecture<T> architecture, GroundingDINOOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Properties
| Property | Summary |
|---|---|
int EmbeddingDimension { get; } |
The width of the image tokens EncodeImage returns. |
int MaxDetections { get; } |
The most detections GroundText returns (the highest scoring ones). |
Methods
| Method | Summary |
|---|---|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
|
IReadOnlyList<GroundingDetection<T>> Ground(Tensor<T> image, string caption) |
The regions the caption’s phrases name (the official predict): each query whose best token probability exceeds the box threshold (ConfidenceThreshold), with that probability, its box as fractions of the image, and the words whose tokens… |
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateGroundingLoss(Tensor<T> image, string caption, IReadOnlyList<GroundingBox> boxes) |
The paper’s loss on an image, its caption and boxes, without training (dropout off). |
Tensor<T> DetectObjects(Tensor<T> image, IReadOnlyList<string> categories) |
The official predict_with_classes: the categories joined as “a. |
Tensor<T> EncodeImage(Tensor<T> image) |
The image tokens before they meet the text: every feature level (the backbone’s and the extra stride-2 level), projected to the model width and group-normalized, flattened and concatenated, [tokens, DecoderDim] (or `[B, tokens, DecoderDim… |
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Tensor<T> GroundText(Tensor<T> image, string textQuery) |
Ground as a tensor [n, 5] of (x1, y1, x2, y2, score) rows, the highest-scoring MaxDetections in query order. |
void SetTrainingMode(bool isTraining) |
|
void TrainGrounding(Tensor<T> image, string caption, IReadOnlyList<GroundingBox> boxes) |
One training step on an image, its caption and the boxes its phrases name, with the paper’s loss (Hungarian matching; contrastive focal, L1 and GIoU losses on every decoder layer and the encoder’s selected queries). |