GLaMMOptions
Options & Configuration · AiDotNet.VisionLanguage.Grounding
Configuration options for GLaMM (Rasheed et al. 2024), the grounding large multimodal model that answers with text and
a segmentation mask for every [SEG] it writes.
Inheritance: ModelOptions → GLaMMOptions
For Beginners
These options configure GLaMM’s four parts: the image encoder, the language model, the region encoder and the segmenter.
How It Works
The defaults are the released model’s (MBZUAI/GLaMM-GranD-Pretrained, built by the official train.py): CLIP ViT-L/14 at 336 px, Vicuna-7B v1.5 with GLaMM’s seven added tokens and LoRA (rank 8, alpha 16, dropout 0.05 on q_proj and v_proj), the region encoder, SAM ViT-H at 1024 px, a 256-wide [SEG] projection, and the loss weights 1 (CE), 2 (BCE) and 0.5 (DICE).
GLaMM reads Vicuna’s SentencePiece model (tokenizer.model, under the Llama 2 license), which is not
shipped: set TokenizerPath or TokenizerModel.
Constructors
| Constructor | Summary |
|---|---|
GLaMMOptions() |
Initializes a new instance with default values. |
GLaMMOptions(GLaMMOptions other) |
Initializes a new instance by copying from another instance. |
Properties
| Property | Summary |
|---|---|
byte[] TokenizerModel { get; set; } |
Gets or sets the bytes of a SentencePiece BPE model, in place of TokenizerPath. |
double BceLossWeight { get; set; } |
Gets or sets the mask cross-entropy weight (2). |
double CeLossWeight { get; set; } |
Gets or sets the language-model loss weight (1). |
double DiceLossWeight { get; set; } |
Gets or sets the mask DICE weight (0.5). |
double LoraAlpha { get; set; } |
Gets or sets LoRA alpha (16). |
double LoraDropout { get; set; } |
Gets or sets LoRA dropout (0.05). |
int ClipHeads { get; set; } |
Gets or sets CLIP’s attention heads (16). |
int ClipHidden { get; set; } |
Gets or sets CLIP’s width (1024). |
int ClipImageSize { get; set; } |
Gets or sets CLIP’s image side (336). |
int ClipIntermediate { get; set; } |
Gets or sets CLIP’s feed-forward width (4096). |
int ClipLayers { get; set; } |
Gets or sets CLIP’s layers (24). |
int ClipPatchSize { get; set; } |
Gets or sets CLIP’s patch side (14). |
int LlmHeads { get; set; } |
Gets or sets the language model’s attention heads (32). |
int LlmHidden { get; set; } |
Gets or sets the language model’s width (4096). |
int LlmIntermediate { get; set; } |
Gets or sets the language model’s SwiGLU width (11008). |
int LlmKeyValueHeads { get; set; } |
Gets or sets the language model’s key-value heads (32). |
int LlmLayers { get; set; } |
Gets or sets the language model’s layers (32). |
int LoraRank { get; set; } |
Gets or sets the LoRA rank on q_proj and v_proj (8); 0 trains every decoder layer instead. |
int ModelMaxLength { get; set; } |
Gets or sets the tokenizer’s maximum length (1536); training keeps the first 1536 − 575 ids, as the official collate does. |
int RegionGroupNormGroups { get; set; } |
Gets or sets the GroupNorm groups of the region encoder’s fuse convolutions (64), which divide ClipHidden. |
int RegionWidth { get; set; } |
Gets or sets the region encoder’s region and position width (1024, fixed in the official code). |
int SamDecoderHeads { get; set; } |
Gets or sets SAM’s mask decoder heads (8). |
int SamDecoderMlpDim { get; set; } |
Gets or sets SAM’s mask decoder feed-forward width (2048). |
int SamEncoderDepth { get; set; } |
Gets or sets SAM’s encoder depth (ViT-H: 32). |
int SamEncoderDim { get; set; } |
Gets or sets SAM’s encoder width (ViT-H: 1280). |
int SamEncoderHeads { get; set; } |
Gets or sets SAM’s encoder heads (16). |
int SamImageSize { get; set; } |
Gets or sets SAM’s image side (1024). |
int SamIouHiddenDim { get; set; } |
Gets or sets SAM’s IoU head width (256). |
int SamMaskInChannels { get; set; } |
Gets or sets the channels of SAM’s mask embedding (16). |
int SamPatchSize { get; set; } |
Gets or sets SAM’s patch side (16). |
int SamPromptDim { get; set; } |
Gets or sets SAM’s prompt and decoder width, also the [SEG] projection’s output (256). |
int SamWindowSize { get; set; } |
Gets or sets SAM’s attention window (14). |
int TokenizerPieces { get; set; } |
Gets or sets the pieces of the tokenizer the language model embeds (32000, Vicuna’s), used to size the network when it is built before its tokenizer is supplied; a tokenizer loaded later must have this many. |
int[] SamGlobalAttention { get; set; } |
Gets or sets SAM’s globally attending blocks (ViT-H: 7, 15, 23, 31). |
string TokenizerPath { get; set; } |
Gets or sets the path of Vicuna’s SentencePiece tokenizer.model. |