MultiModalEmbeddingModel<T>
Models & Types · AiDotNet.RetrievalAugmentedGeneration.EmbeddingModels
Multi-modal embedding model that creates unified vector representations for both text and images in the same embedding space.
Inheritance: EmbeddingModelBase<T> → MultiModalEmbeddingModel<T>
For Beginners
Imagine a search engine where you can find pictures using words, or vice versa.
Normal embeddings:
- Text embedding: “a cat” → [0.2, 0.5, 0.1, …]
- Image embedding: (cat.jpg) → [0.9, 0.1, 0.3, …] (different space, can’t compare!)
Multi-modal embeddings (same space):
- Text: “a cat” → [0.2, 0.5, 0.1, …]
- Image: (cat.jpg) → [0.21, 0.48, 0.12, …] (similar values = same meaning!)
- Result: Can calculate similarity between text and image!
Real-world uses:
- Search images with text: “sunset over mountains” finds mountain sunset photos
- Search text with images: Upload product photo, find product descriptions
- Visual Q&A: “What’s in this image?” matched against image embeddings
- Content recommendation: Show images similar to text user likes
How it works: The model was trained with millions of (image, text) pairs to learn that:
- Image of dog + text “dog” should have similar embeddings
- Image of dog + text “cat” should have different embeddings
- This creates a “shared understanding” between vision and language
How It Works
This model enables cross-modal similarity search by embedding text and images into the same vector space, allowing you to search for images using text queries, find similar images, or retrieve text documents related to images. Based on models like CLIP (Contrastive Language-Image Pre-training).
Example Usage:
How It Works: Architecture (simplified CLIP):
Text Path:
- Text → Tokenizer → Token IDs
- Token IDs → Text Encoder (Transformer) → Text Features
- Text Features → Projection Head → Text Embedding (512D)
Image Path:
- Image → Resize/Normalize → Pixel Array
- Pixels → Image Encoder (CNN/Vision Transformer) → Image Features
- Image Features → Projection Head → Image Embedding (512D)
Both paths output embeddings in the same 512-dimensional space, enabling direct comparison.
Current implementation uses ONNX for text encoding only. The image path
(EmbedImage) is not implemented and throws NotSupportedException
rather than fabricating a vector; a full CLIP ONNX vision encoder must be connected for it.
Benefits:
- Cross-modal search - Find images with text, text with images
- Zero-shot classification - Classify images without task-specific training
- Unified representation - Single embedding space for all content types
- Flexible retrieval - Mix text and image documents in same index
- Pre-trained - No custom training needed for basic use cases
Limitations:
- Image encoding is NOT implemented (EmbedImage throws NotSupportedException) - needs real CLIP encoder
- Model file size can be large (1-4GB for CLIP variants)
- Image processing slower than text (convolutional layers are compute-intensive)
- Works best with natural images (struggles with abstract diagrams, charts)
- Limited to English text in most CLIP models (multilingual versions available)
Constructors
| Constructor | Summary |
|---|---|
MultiModalEmbeddingModel(string modelPath, bool normalizeEmbeddings, int dimension) |
Initializes a new instance of the MultiModalEmbeddingModel class. |
Properties
| Property | Summary |
|---|---|
int EmbeddingDimension { get; } |
|
int MaxTokens { get; } |
Methods
| Method | Summary |
|---|---|
IEnumerable<Vector<T>> EmbedImageBatch(IEnumerable<string> imagePaths) |
Batch embedding generation for images. |
Vector<T> EmbedImage(string imagePath) |
Generates image embeddings from a file path. |