LLM2CLIP<T>

Models & Types · AiDotNet.VisionLanguage.Encoders

LLM2CLIP model enhancing CLIP’s text encoder with LLM embeddings for richer semantic understanding.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → LLM2CLIP<T>

Implements: IContrastiveVisionLanguageModel<T>

For Beginners

LLM2CLIP supercharges CLIP by replacing its simple text encoder with a powerful large language model (like LLaMA or Mistral). This gives the model much richer text understanding — it can handle complex descriptions, long captions, and nuanced queries that the original CLIP text encoder would struggle with. Default values follow the original paper settings.

How It Works

LLM2CLIP (Huang et al., 2024) replaces CLIP’s text encoder with a fine-tuned LLM (e.g., LLaMA, Mistral), providing much stronger text understanding for complex captions and long-form descriptions.

References:

  • Paper: “LLM2CLIP: Powerful Language Model Unlock Richer Visual Representation” (Huang et al., 2024)

Example

var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
var trainModel = new LLM2CLIP<double>(architecture, new LLM2CLIPOptions());

Constructors

Constructor Summary
LLM2CLIP(NeuralNetworkArchitecture<T> architecture, LLM2CLIPOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
LLM2CLIP(NeuralNetworkArchitecture<T> architecture, string imageEncoderModelPath, LLM2CLIPOptions options)

Properties

Property Summary
T Temperature { get; }
int EmbeddingDimension { get; }
int MaxSequenceLength { get; }
int ProjectionDimension { get; }
int TextEmbeddingDimension { get; }

Methods

Method Summary
Dictionary<string, T> ZeroShotClassify(Tensor<T> image, string[] labels)
ModelMetadata<T> GetModelMetadata()
T ComputeSimilarity(Tensor<T> image, string text)
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> EncodeText(string text)
Tensor<T>[] EncodeTexts(string[] texts)