LiT<T>

Models & Types · AiDotNet.VisionLanguage.Encoders

LiT (Locked-image Tuning) model that freezes a pre-trained image encoder and only trains the text encoder for efficient contrastive image-text alignment.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → LiT<T>

Implements: IContrastiveVisionLanguageModel<T>

For Beginners

LiT makes CLIP training much cheaper by freezing a pre-trained image encoder and only training the text encoder to align with it. Instead of training both vision and language components from scratch, it reuses an existing strong vision model and teaches a text model to “read” its image features, achieving 85.2% zero-shot ImageNet accuracy with much less compute. Default values follow the original paper settings.

How It Works

LiT (Zhai et al., CVPR 2022) demonstrates that contrastive image-text training can be made dramatically more efficient by freezing a high-quality pre-trained vision encoder (e.g., from ImageNet-21k) and only training the text encoder to align with it. This “locked-image tuning” achieves 85.2% zero-shot accuracy on ImageNet with significantly reduced training cost.

References:

  • Paper: “LiT: Zero-Shot Transfer with Locked-image text Tuning” (Zhai et al., CVPR 2022)

Example

var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
var trainModel = new LiT<double>(architecture, new LiTOptions());

Constructors

Constructor Summary
LiT(NeuralNetworkArchitecture<T> architecture, LiTOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
LiT(NeuralNetworkArchitecture<T> architecture, string imageEncoderModelPath, LiTOptions options)

Properties

Property Summary
T Temperature { get; }
int EmbeddingDimension { get; }
int MaxSequenceLength { get; }
int ProjectionDimension { get; }
int TextEmbeddingDimension { get; }

Methods

Method Summary
Dictionary<string, T> ZeroShotClassify(Tensor<T> image, string[] labels)
ModelMetadata<T> GetModelMetadata()
T ComputeSimilarity(Tensor<T> image, string text)
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> EncodeText(string text)
Tensor<T>[] EncodeTexts(string[] texts)