GLaMM<T>

Models & Types · AiDotNet.VisionLanguage.Grounding

GLaMM: Pixel Grounding Large Multimodal Model.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → GLaMM<T>

Implements: IGroundedConversationTrainable<T>

For Beginners

GLaMM chats about an image and draws a mask on it for each object it mentions.

How It Works

GLaMM (Rasheed et al., 2024) is LLaVA-1.5 (a CLIP ViT-L/14-336 image encoder, a GELU projector and Vicuna-7B) with a region encoder, so a question can point at boxes (<bbox>), and SAM ViT-H, so the answer can point at pixels: every [SEG] the language model writes is decoded into a mask from that token’s hidden state. It is built as the official code builds it (mbzuai-oryx/groundingLMM), and one training sample matches the official computation to 1e-9.

Chat runs the official app: the instruction after “The provides an overview of the picture.”, greedy decoding (the released checkpoints set no sampling), and the masks of the [SEG] tokens. SegmentExpression, DescribeGrounded and CaptionRegion use the official referring-segmentation, grounded-caption and region-caption prompts. Training takes conversations with masks (TrainConversation): the language-model loss plus 2 x mask cross-entropy plus 0.5 x DICE, updating the modules the official training updates (the token embeddings and head, LoRA on q_proj and v_proj, the region encoder, the [SEG] projection and SAM’s mask decoder).

Images are read in 0..1 at any size. As the official processors do, CLIP sees the shorter side resized to 336 (bicubic) and centre-cropped, and SAM the longer side resized to 1024 (bilinear) and zero-padded. Masks come back at the image’s size.

References:

  • Paper: “GLaMM: Pixel Grounding Large Multimodal Model” (Rasheed et al., CVPR 2024)

Constructors

Constructor Summary
GLaMM(NeuralNetworkArchitecture<T> architecture, GLaMMOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)

Methods

Method Summary
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
GlammAnswer<T> Chat(Tensor<T> image, string instruction, IReadOnlyList<double[]> regions, int maxNewTokens) One answer as the official app produces it: the instruction (which may name regions with <bbox>, one per entry of regions), greedy decoding until or maxNewTokens, the text after the last “ASSISTANT: “, and a mask for each [SEG]…
GlammAnswer<T> DescribeGrounded(Tensor<T> image, int maxNewTokens) Grounded conversation generation with the official prompt: a detailed description whose phrases (<p> … </p>) each have a mask.
ModelMetadata<T> GetModelMetadata()
T EvaluateConversationLoss(GlammSample<T> sample) GLaMM’s loss on a sample, without training (LoRA dropout off).
Tensor<T> EncodeImage(Tensor<T> image) The projected CLIP patch features, LLaVA’s visual tokens, of images in 0..1: [Patches, LlmHidden] for one image [3, H, W], [B, Patches, LlmHidden] for a batch [B, 3, H, W].
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> SegmentExpression(Tensor<T> image, string expression) Referring segmentation with the official prompt (“Can you segment the {expression} in this image?”): the union of the answer’s masks [H, W] (zero when it wrote no [SEG]).
string CaptionRegion(Tensor<T> image, double[] box, int maxNewTokens) Region captioning with the official prompt, for one box (x1, y1, x2, y2) as fractions of the image.
void SetTrainingMode(bool isTraining)
void TrainConversation(GlammSample<T> sample) One training step on a conversation with masks, with GLaMM’s loss and the official trainable modules.