LLM2CLIPOptions

Options & Configuration · AiDotNet.VisionLanguage.Encoders

Configuration options for the LLM2CLIP model.

Inheritance: ModelOptions → ContrastiveEncoderOptions → LLM2CLIPOptions

For Beginners

LLM2CLIP upgrades CLIP’s text understanding by replacing its simple text model with a powerful language model (like ChatGPT’s text engine). This means it can understand complex descriptions, nuanced language, and longer text much better than regular CLIP.

How It Works

LLM2CLIP (Huang et al., 2024) from Microsoft enhances CLIP’s text encoder by replacing it with an LLM (such as LLaMA or Mistral). The LLM text embeddings provide richer semantic understanding, especially for complex captions and long-form text. The LLM is fine-tuned with contrastive learning to align with the existing CLIP vision encoder.

Constructors

Constructor Summary
LLM2CLIPOptions() Initializes default LLM2CLIP options.
LLM2CLIPOptions(LLM2CLIPOptions other) Initializes a new instance by copying from another instance.

Properties

Property Summary
ContrastiveLossType LossType { get; set; } Gets or sets the contrastive loss type.
bool FreezeVisionEncoder { get; set; } Gets or sets whether to freeze the vision encoder during LLM alignment.
bool UseLoRA { get; set; } Gets or sets whether to use LoRA for efficient LLM fine-tuning.
int LLMHiddenDim { get; set; } Gets or sets the LLM hidden dimension.
int LoRARank { get; set; } Gets or sets the LoRA rank for efficient fine-tuning.
string LLMBackbone { get; set; } Gets or sets the LLM backbone name for the text encoder.