Groma<T>
Models & Types · AiDotNet.VisionLanguage.Grounding
Groma: localized visual tokenization for grounded understanding.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → Groma<T>
Implements: IVisualGroundingModel<T>
For Beginners
Groma is a vision-language model that grounds language understanding in specific image regions using localized visual tokens. Default values follow the original paper settings.
How It Works
Groma (Ma et al., 2024) introduces localized visual tokenization for grounding multimodal LLMs. It uses a detect-then-describe pipeline with a region proposal network to extract candidate object regions, RoI-Align for fixed-size region feature extraction, and quantized region tokens interleaved with text tokens, enabling the LLM to refer to and describe specific image regions without requiring explicit coordinate output in the text stream.
References:
- Paper: “Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models” (ByteDance, 2024)
Example
// Create a Groma model for localized visual grounding
// with region-level visual tokenization for region-text alignment
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
// ONNX inference mode with pre-trained model
var model = new Groma<double>(architecture, "groma.onnx");
// Training mode with native layers
var trainModel = new Groma<double>(architecture, new GromaOptions());
Constructors
| Constructor | Summary |
|---|---|
Groma(NeuralNetworkArchitecture<T> architecture, GromaOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
Groma(NeuralNetworkArchitecture<T> architecture, string modelPath, GromaOptions options) |
Properties
| Property | Summary |
|---|---|
int EmbeddingDimension { get; } |
|
int MaxDetections { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> DetectObjects(Tensor<T> image, IReadOnlyList<string> categories) |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> GroundText(Tensor<T> image, string textQuery) |
Grounds text using Groma’s localized visual tokenization approach. |