LiT<T>
Models & Types · AiDotNet.VisionLanguage.Encoders
LiT (Locked-image Tuning) model that freezes a pre-trained image encoder and only trains the text encoder for efficient contrastive image-text alignment.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → LiT<T>
Implements: IContrastiveVisionLanguageModel<T>
For Beginners
LiT makes CLIP training much cheaper by freezing a pre-trained image encoder and only training the text encoder to align with it. Instead of training both vision and language components from scratch, it reuses an existing strong vision model and teaches a text model to “read” its image features, achieving 85.2% zero-shot ImageNet accuracy with much less compute. Default values follow the original paper settings.
How It Works
LiT (Zhai et al., CVPR 2022) demonstrates that contrastive image-text training can be made dramatically more efficient by freezing a high-quality pre-trained vision encoder (e.g., from ImageNet-21k) and only training the text encoder to align with it. This “locked-image tuning” achieves 85.2% zero-shot accuracy on ImageNet with significantly reduced training cost.
References:
- Paper: “LiT: Zero-Shot Transfer with Locked-image text Tuning” (Zhai et al., CVPR 2022)
Example
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
var trainModel = new LiT<double>(architecture, new LiTOptions());
Constructors
| Constructor | Summary |
|---|---|
LiT(NeuralNetworkArchitecture<T> architecture, LiTOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
LiT(NeuralNetworkArchitecture<T> architecture, string imageEncoderModelPath, LiTOptions options) |
Properties
| Property | Summary |
|---|---|
T Temperature { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxSequenceLength { get; } |
|
int ProjectionDimension { get; } |
|
int TextEmbeddingDimension { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> ZeroShotClassify(Tensor<T> image, string[] labels) |
|
ModelMetadata<T> GetModelMetadata() |
|
T ComputeSimilarity(Tensor<T> image, string text) |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> EncodeText(string text) |
|
Tensor<T>[] EncodeTexts(string[] texts) |