GLaMM<T>
Models & Types · AiDotNet.VisionLanguage.Grounding
GLaMM: Pixel Grounding Large Multimodal Model.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → GLaMM<T>
Implements: IGroundedConversationTrainable<T>
For Beginners
GLaMM chats about an image and draws a mask on it for each object it mentions.
How It Works
GLaMM (Rasheed et al., 2024) is LLaVA-1.5 (a CLIP ViT-L/14-336 image encoder, a GELU projector and Vicuna-7B) with a
region encoder, so a question can point at boxes (<bbox>), and SAM ViT-H, so the answer can point at
pixels: every [SEG] the language model writes is decoded into a mask from that token’s hidden state. It is
built as the official code builds it (mbzuai-oryx/groundingLMM), and one training sample matches the official
computation to 1e-9.
Chat runs the official app: the instruction after “The SegmentExpression, DescribeGrounded and CaptionRegion use the official
referring-segmentation, grounded-caption and region-caption prompts. Training takes conversations with masks
(TrainConversation): the language-model loss plus 2 x mask cross-entropy plus 0.5 x DICE, updating the
modules the official training updates (the token embeddings and head, LoRA on q_proj and v_proj, the region encoder,
the [SEG] projection and SAM’s mask decoder).
Images are read in 0..1 at any size. As the official processors do, CLIP sees the shorter side resized to 336 (bicubic) and centre-cropped, and SAM the longer side resized to 1024 (bilinear) and zero-padded. Masks come back at the image’s size.
References:
- Paper: “GLaMM: Pixel Grounding Large Multimodal Model” (Rasheed et al., CVPR 2024)
Constructors
| Constructor | Summary |
|---|---|
GLaMM(NeuralNetworkArchitecture<T> architecture, GLaMMOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Methods
| Method | Summary |
|---|---|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
|
GlammAnswer<T> Chat(Tensor<T> image, string instruction, IReadOnlyList<double[]> regions, int maxNewTokens) |
One answer as the official app produces it: the instruction (which may name regions with <bbox>, one per entry of regions), greedy decoding until or maxNewTokens, the text after the last “ASSISTANT: “, and a mask for each [SEG]… |
GlammAnswer<T> DescribeGrounded(Tensor<T> image, int maxNewTokens) |
Grounded conversation generation with the official prompt: a detailed description whose phrases (<p> … </p>) each have a mask. |
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateConversationLoss(GlammSample<T> sample) |
GLaMM’s loss on a sample, without training (LoRA dropout off). |
Tensor<T> EncodeImage(Tensor<T> image) |
The projected CLIP patch features, LLaVA’s visual tokens, of images in 0..1: [Patches, LlmHidden] for one image [3, H, W], [B, Patches, LlmHidden] for a batch [B, 3, H, W]. |
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Tensor<T> SegmentExpression(Tensor<T> image, string expression) |
Referring segmentation with the official prompt (“Can you segment the {expression} in this image?”): the union of the answer’s masks [H, W] (zero when it wrote no [SEG]). |
string CaptionRegion(Tensor<T> image, double[] box, int maxNewTokens) |
Region captioning with the official prompt, for one box (x1, y1, x2, y2) as fractions of the image. |
void SetTrainingMode(bool isTraining) |
|
void TrainConversation(GlammSample<T> sample) |
One training step on a conversation with masks, with GLaMM’s loss and the official trainable modules. |