GLaMMOptions

Options & Configuration · AiDotNet.VisionLanguage.Grounding

Configuration options for GLaMM (Rasheed et al. 2024), the grounding large multimodal model that answers with text and a segmentation mask for every [SEG] it writes.

Inheritance: ModelOptions → GLaMMOptions

For Beginners

These options configure GLaMM’s four parts: the image encoder, the language model, the region encoder and the segmenter.

How It Works

The defaults are the released model’s (MBZUAI/GLaMM-GranD-Pretrained, built by the official train.py): CLIP ViT-L/14 at 336 px, Vicuna-7B v1.5 with GLaMM’s seven added tokens and LoRA (rank 8, alpha 16, dropout 0.05 on q_proj and v_proj), the region encoder, SAM ViT-H at 1024 px, a 256-wide [SEG] projection, and the loss weights 1 (CE), 2 (BCE) and 0.5 (DICE).

GLaMM reads Vicuna’s SentencePiece model (tokenizer.model, under the Llama 2 license), which is not shipped: set TokenizerPath or TokenizerModel.

Constructors

Constructor Summary
GLaMMOptions() Initializes a new instance with default values.
GLaMMOptions(GLaMMOptions other) Initializes a new instance by copying from another instance.

Properties

Property Summary
byte[] TokenizerModel { get; set; } Gets or sets the bytes of a SentencePiece BPE model, in place of TokenizerPath.
double BceLossWeight { get; set; } Gets or sets the mask cross-entropy weight (2).
double CeLossWeight { get; set; } Gets or sets the language-model loss weight (1).
double DiceLossWeight { get; set; } Gets or sets the mask DICE weight (0.5).
double LoraAlpha { get; set; } Gets or sets LoRA alpha (16).
double LoraDropout { get; set; } Gets or sets LoRA dropout (0.05).
int ClipHeads { get; set; } Gets or sets CLIP’s attention heads (16).
int ClipHidden { get; set; } Gets or sets CLIP’s width (1024).
int ClipImageSize { get; set; } Gets or sets CLIP’s image side (336).
int ClipIntermediate { get; set; } Gets or sets CLIP’s feed-forward width (4096).
int ClipLayers { get; set; } Gets or sets CLIP’s layers (24).
int ClipPatchSize { get; set; } Gets or sets CLIP’s patch side (14).
int LlmHeads { get; set; } Gets or sets the language model’s attention heads (32).
int LlmHidden { get; set; } Gets or sets the language model’s width (4096).
int LlmIntermediate { get; set; } Gets or sets the language model’s SwiGLU width (11008).
int LlmKeyValueHeads { get; set; } Gets or sets the language model’s key-value heads (32).
int LlmLayers { get; set; } Gets or sets the language model’s layers (32).
int LoraRank { get; set; } Gets or sets the LoRA rank on q_proj and v_proj (8); 0 trains every decoder layer instead.
int ModelMaxLength { get; set; } Gets or sets the tokenizer’s maximum length (1536); training keeps the first 1536 − 575 ids, as the official collate does.
int RegionGroupNormGroups { get; set; } Gets or sets the GroupNorm groups of the region encoder’s fuse convolutions (64), which divide ClipHidden.
int RegionWidth { get; set; } Gets or sets the region encoder’s region and position width (1024, fixed in the official code).
int SamDecoderHeads { get; set; } Gets or sets SAM’s mask decoder heads (8).
int SamDecoderMlpDim { get; set; } Gets or sets SAM’s mask decoder feed-forward width (2048).
int SamEncoderDepth { get; set; } Gets or sets SAM’s encoder depth (ViT-H: 32).
int SamEncoderDim { get; set; } Gets or sets SAM’s encoder width (ViT-H: 1280).
int SamEncoderHeads { get; set; } Gets or sets SAM’s encoder heads (16).
int SamImageSize { get; set; } Gets or sets SAM’s image side (1024).
int SamIouHiddenDim { get; set; } Gets or sets SAM’s IoU head width (256).
int SamMaskInChannels { get; set; } Gets or sets the channels of SAM’s mask embedding (16).
int SamPatchSize { get; set; } Gets or sets SAM’s patch side (16).
int SamPromptDim { get; set; } Gets or sets SAM’s prompt and decoder width, also the [SEG] projection’s output (256).
int SamWindowSize { get; set; } Gets or sets SAM’s attention window (14).
int TokenizerPieces { get; set; } Gets or sets the pieces of the tokenizer the language model embeds (32000, Vicuna’s), used to size the network when it is built before its tokenizer is supplied; a tokenizer loaded later must have this many.
int[] SamGlobalAttention { get; set; } Gets or sets SAM’s globally attending blocks (ViT-H: 7, 15, 23, 31).
string TokenizerPath { get; set; } Gets or sets the path of Vicuna’s SentencePiece tokenizer.model.