IMultimodalEmbedding<T>
Interfaces · AiDotNet.Interfaces
Interface for multimodal embedding models that can encode multiple modalities (text, images, audio) into a shared embedding space.
Known implementations: ClipNeuralNetwork<T>, IBlip2Model<T>, IBlipModel<T>, IFlamingoModel<T>, IGpt4VisionModel<T>, ILLaVAModel<T>, IVideoCLIPModel<T>
For Beginners
Imagine you want to search for images using text queries. A multimodal model learns to convert both “a photo of a cat” and an actual cat image into similar vectors, allowing direct comparison between text and images.
How It Works
Multimodal embedding models like CLIP (Contrastive Language-Image Pre-training) learn to project different types of data into the same vector space, enabling cross-modal similarity search and zero-shot classification.
Properties
| Property | Summary |
|---|---|
int EmbeddingDimension { get; } |
Gets the dimensionality of the embedding space. |
int ImageSize { get; } |
Gets the expected image size (square images: ImageSize x ImageSize pixels). |
int MaxSequenceLength { get; } |
Gets the maximum sequence length for text input. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> ZeroShotClassify(double[] imageData, IEnumerable<string> labels) |
Performs zero-shot classification of an image against text labels. |
Matrix<T> EncodeImageBatch(IEnumerable<double[]> imageDataBatch) |
Encodes multiple images into embedding vectors in a batch. |
Matrix<T> EncodeTextBatch(IEnumerable<string> texts) |
Encodes multiple texts into embedding vectors in a batch. |
T ComputeSimilarity(Vector<T> embedding1, Vector<T> embedding2) |
Computes similarity between two embeddings. |
Vector<T> EncodeImage(double[] imageData) |
Encodes an image into an embedding vector. |
Vector<T> EncodeText(string text) |
Encodes text into an embedding vector. |