IGroundingTrainable<T>

Interfaces · AiDotNet.VisionLanguage.Interfaces

A grounding model that trains on grounded phrases: an image, a caption, and the boxes the caption’s phrases name.

Known implementations: Ferret<T>, GroundedSAM2<T>, GroundingDINO<T>, Shikra<T>

How It Works

Such a model’s paper loss matches its predictions to the boxes and scores them against the caption’s tokens. A tensor target cannot carry the phrases, so these models refuse Train(input, expected) and train here instead.

Methods

Method Summary
T EvaluateGroundingLoss(Tensor<T> image, string caption, IReadOnlyList<GroundingBox> boxes) The paper loss on the same supervision, without training.
void TrainGrounding(Tensor<T> image, string caption, IReadOnlyList<GroundingBox> boxes) One training step with the model’s paper loss.