IMultimodalEmbedding<T>

Interfaces · AiDotNet.Interfaces

Interface for multimodal embedding models that can encode multiple modalities (text, images, audio) into a shared embedding space.

Known implementations: ClipNeuralNetwork<T>, IBlip2Model<T>, IBlipModel<T>, IFlamingoModel<T>, IGpt4VisionModel<T>, ILLaVAModel<T>, IVideoCLIPModel<T>

For Beginners

Imagine you want to search for images using text queries. A multimodal model learns to convert both “a photo of a cat” and an actual cat image into similar vectors, allowing direct comparison between text and images.

How It Works

Multimodal embedding models like CLIP (Contrastive Language-Image Pre-training) learn to project different types of data into the same vector space, enabling cross-modal similarity search and zero-shot classification.

Properties

Property Summary
int EmbeddingDimension { get; } Gets the dimensionality of the embedding space.
int ImageSize { get; } Gets the expected image size (square images: ImageSize x ImageSize pixels).
int MaxSequenceLength { get; } Gets the maximum sequence length for text input.

Methods

Method Summary
Dictionary<string, T> ZeroShotClassify(double[] imageData, IEnumerable<string> labels) Performs zero-shot classification of an image against text labels.
Matrix<T> EncodeImageBatch(IEnumerable<double[]> imageDataBatch) Encodes multiple images into embedding vectors in a batch.
Matrix<T> EncodeTextBatch(IEnumerable<string> texts) Encodes multiple texts into embedding vectors in a batch.
T ComputeSimilarity(Vector<T> embedding1, Vector<T> embedding2) Computes similarity between two embeddings.
Vector<T> EncodeImage(double[] imageData) Encodes an image into an embedding vector.
Vector<T> EncodeText(string text) Encodes text into an embedding vector.