LLM2CLIP<T>
Models & Types · AiDotNet.VisionLanguage.Encoders
LLM2CLIP model enhancing CLIP’s text encoder with LLM embeddings for richer semantic understanding.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → LLM2CLIP<T>
Implements: IContrastiveVisionLanguageModel<T>
For Beginners
LLM2CLIP supercharges CLIP by replacing its simple text encoder with a powerful large language model (like LLaMA or Mistral). This gives the model much richer text understanding — it can handle complex descriptions, long captions, and nuanced queries that the original CLIP text encoder would struggle with. Default values follow the original paper settings.
How It Works
LLM2CLIP (Huang et al., 2024) replaces CLIP’s text encoder with a fine-tuned LLM (e.g., LLaMA, Mistral), providing much stronger text understanding for complex captions and long-form descriptions.
References:
- Paper: “LLM2CLIP: Powerful Language Model Unlock Richer Visual Representation” (Huang et al., 2024)
Example
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
var trainModel = new LLM2CLIP<double>(architecture, new LLM2CLIPOptions());
Constructors
| Constructor | Summary |
|---|---|
LLM2CLIP(NeuralNetworkArchitecture<T> architecture, LLM2CLIPOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
LLM2CLIP(NeuralNetworkArchitecture<T> architecture, string imageEncoderModelPath, LLM2CLIPOptions options) |
Properties
| Property | Summary |
|---|---|
T Temperature { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxSequenceLength { get; } |
|
int ProjectionDimension { get; } |
|
int TextEmbeddingDimension { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> ZeroShotClassify(Tensor<T> image, string[] labels) |
|
ModelMetadata<T> GetModelMetadata() |
|
T ComputeSimilarity(Tensor<T> image, string text) |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> EncodeText(string text) |
|
Tensor<T>[] EncodeTexts(string[] texts) |