Groma<T>

Models & Types · AiDotNet.VisionLanguage.Grounding

Groma: localized visual tokenization for grounded understanding.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → Groma<T>

Implements: IVisualGroundingModel<T>

For Beginners

Groma is a vision-language model that grounds language understanding in specific image regions using localized visual tokens. Default values follow the original paper settings.

How It Works

Groma (Ma et al., 2024) introduces localized visual tokenization for grounding multimodal LLMs. It uses a detect-then-describe pipeline with a region proposal network to extract candidate object regions, RoI-Align for fixed-size region feature extraction, and quantized region tokens interleaved with text tokens, enabling the LLM to refer to and describe specific image regions without requiring explicit coordinate output in the text stream.

References:

  • Paper: “Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models” (ByteDance, 2024)

Example

// Create a Groma model for localized visual grounding
// with region-level visual tokenization for region-text alignment
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new Groma<double>(architecture, "groma.onnx");
            
// Training mode with native layers
var trainModel = new Groma<double>(architecture, new GromaOptions());

Constructors

Constructor Summary
Groma(NeuralNetworkArchitecture<T> architecture, GromaOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
Groma(NeuralNetworkArchitecture<T> architecture, string modelPath, GromaOptions options)

Properties

Property Summary
int EmbeddingDimension { get; }
int MaxDetections { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> DetectObjects(Tensor<T> image, IReadOnlyList<string> categories)
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GroundText(Tensor<T> image, string textQuery) Grounds text using Groma’s localized visual tokenization approach.