IImageBindModel<T>
Interfaces · AiDotNet.Interfaces
Defines the contract for ImageBind models that bind multiple modalities (6+) into a shared embedding space.
Known implementations: ImageBindNeuralNetwork<T>
For Beginners
ImageBind connects ALL types of data together!
The breakthrough insight:
- Images are paired with many things: text captions, video audio, depth sensors, etc.
- By learning all these (image, X) pairs, images become a “bridge”
- Now audio and text can be compared, even without audio-text training data!
Six modalities in one model:
- Images: Regular RGB photos
- Text: Natural language descriptions
- Audio: Sound waveforms (speech, music, effects)
- Video: Moving images (sequences of frames)
- Thermal: Heat maps from infrared cameras
- Depth: 3D distance information
- IMU: Motion sensor data (accelerometer, gyroscope)
Why this matters:
- Search audio by describing sounds in text
- Find images that match a piece of music
- Match thermal images to regular photos
- Universal multimodal understanding!
How It Works
ImageBind learns a joint embedding space across six modalities: images, text, audio, depth, thermal, and IMU data. It uses images as a binding modality - since web data contains many (image, text) pairs, (image, audio) pairs from videos, etc., the model can learn cross-modal relationships even without direct pairs between all modalities.
Properties
| Property | Summary |
|---|---|
IReadOnlyList<ModalityType> SupportedModalities { get; } |
Gets the list of supported modalities. |
int EmbeddingDimension { get; } |
Gets the dimensionality of the shared embedding space. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> ZeroShotClassify(ModalityType modality, object data, IEnumerable<string> classLabels) |
Performs zero-shot classification across modalities. |
IEnumerable<ValueTuple<int, T>> CrossModalRetrieval(Vector<T> queryEmbedding, IEnumerable<Vector<T>> targetEmbeddings, int topK) |
Performs cross-modal retrieval from one modality to another. |
IEnumerable<ValueTuple<string, T>> GenerateDescriptions(ModalityType modality, object data, IEnumerable<string> candidateDescriptions, int topK) |
Generates text description for non-text modalities. |
T ComputeCrossModalSimilarity(Vector<T> embedding1, Vector<T> embedding2) |
Computes similarity between embeddings from any two modalities. |
T ComputeEmergentAudioTextSimilarity(Tensor<T> audio, string text) |
Computes emergent cross-modal relationships without explicit pairing. |
ValueTuple<ModalityType, object, T> FindBestMatch(ModalityType queryModality, object queryData, IEnumerable<ValueTuple<ModalityType, object>> candidates) |
Finds the best matching modality representation for a query. |
ValueTuple<T, Dictionary<string, object>> ComputeAlignment(ModalityType modality1, object data1, ModalityType modality2, object data2) |
Computes the alignment between two modalities given paired data. |
Vector<T> FuseModalities(Dictionary<ModalityType, Vector<T>> modalityEmbeddings, string fusionMethod) |
Performs multimodal fusion by combining embeddings from multiple modalities. |
Vector<T> GetAudioEmbedding(Tensor<T> audioWaveform, int sampleRate) |
Converts audio into a shared embedding vector. |
Vector<T> GetDepthEmbedding(Tensor<T> depthMap) |
Converts depth map into a shared embedding vector. |
Vector<T> GetEmbedding(ModalityType modality, object data) |
Gets embedding for any supported modality using a generic interface. |
Vector<T> GetIMUEmbedding(Tensor<T> imuData) |
Converts IMU sensor data into a shared embedding vector. |
Vector<T> GetImageEmbedding(Tensor<T> image) |
Converts an image into a shared embedding vector. |
Vector<T> GetTextEmbedding(string text) |
Converts text into a shared embedding vector. |
Vector<T> GetThermalEmbedding(Tensor<T> thermalImage) |
Converts thermal image into a shared embedding vector. |
Vector<T> GetVideoEmbedding(IEnumerable<Tensor<T>> frames) |
Converts video into a shared embedding vector. |