IContrastiveVisionLanguageModel<T>

Interfaces · AiDotNet.VisionLanguage.Interfaces

Interface for contrastive vision-language models that align image and text embeddings in a shared space for zero-shot classification and cross-modal retrieval.

Implements: ITextEncoder<T>, IVisualEncoder<T>

Known implementations: ALIGN<T>, BASIC<T>, BiomedCLIP<T>, CLIPA<T>, DFNCLIP<T>, DeCLIP<T>, EVACLIP<T>, FLIP<T>, LLM2CLIP<T>, LiT<T>, MedCLIP<T>, MetaCLIP<T>, OpenCLIP<T>, RegionCLIP<T>, RemoteCLIP<T>, SigLIP2<T>, SigLIP<T>

For Beginners

Think of this like a universal translator between images and text. The model learns that a photo of a dog and the text “a photo of a dog” should have similar representations. This means you can search for images using text, or classify images by comparing them with text descriptions - without training on specific categories.

How It Works

Contrastive VLMs (like CLIP, SigLIP, ALIGN) learn to place matching image-text pairs close together in a shared embedding space while pushing non-matching pairs apart. This enables:

  • Zero-shot classification: Classify images using text descriptions without training
  • Image-text retrieval: Find images matching a text query, or texts describing an image
  • Similarity scoring: Measure how well an image matches a text description

Properties

Property Summary
T Temperature { get; } Gets the temperature parameter used for similarity scaling.
int ProjectionDimension { get; } Gets the dimensionality of the shared projection space.

Methods

Method Summary
Dictionary<string, T> ZeroShotClassify(Tensor<T> image, string[] labels) Performs zero-shot image classification by comparing the image with text labels.
T ComputeSimilarity(Tensor<T> image, string text) Computes the similarity score between an image and a text description.