IImageBindModel<T>

Interfaces · AiDotNet.Interfaces

Defines the contract for ImageBind models that bind multiple modalities (6+) into a shared embedding space.

Known implementations: ImageBindNeuralNetwork<T>

For Beginners

ImageBind connects ALL types of data together!

The breakthrough insight:

  • Images are paired with many things: text captions, video audio, depth sensors, etc.
  • By learning all these (image, X) pairs, images become a “bridge”
  • Now audio and text can be compared, even without audio-text training data!

Six modalities in one model:

  1. Images: Regular RGB photos
  2. Text: Natural language descriptions
  3. Audio: Sound waveforms (speech, music, effects)
  4. Video: Moving images (sequences of frames)
  5. Thermal: Heat maps from infrared cameras
  6. Depth: 3D distance information
  7. IMU: Motion sensor data (accelerometer, gyroscope)

Why this matters:

  • Search audio by describing sounds in text
  • Find images that match a piece of music
  • Match thermal images to regular photos
  • Universal multimodal understanding!

How It Works

ImageBind learns a joint embedding space across six modalities: images, text, audio, depth, thermal, and IMU data. It uses images as a binding modality - since web data contains many (image, text) pairs, (image, audio) pairs from videos, etc., the model can learn cross-modal relationships even without direct pairs between all modalities.

Properties

Property Summary
IReadOnlyList<ModalityType> SupportedModalities { get; } Gets the list of supported modalities.
int EmbeddingDimension { get; } Gets the dimensionality of the shared embedding space.

Methods

Method Summary
Dictionary<string, T> ZeroShotClassify(ModalityType modality, object data, IEnumerable<string> classLabels) Performs zero-shot classification across modalities.
IEnumerable<ValueTuple<int, T>> CrossModalRetrieval(Vector<T> queryEmbedding, IEnumerable<Vector<T>> targetEmbeddings, int topK) Performs cross-modal retrieval from one modality to another.
IEnumerable<ValueTuple<string, T>> GenerateDescriptions(ModalityType modality, object data, IEnumerable<string> candidateDescriptions, int topK) Generates text description for non-text modalities.
T ComputeCrossModalSimilarity(Vector<T> embedding1, Vector<T> embedding2) Computes similarity between embeddings from any two modalities.
T ComputeEmergentAudioTextSimilarity(Tensor<T> audio, string text) Computes emergent cross-modal relationships without explicit pairing.
ValueTuple<ModalityType, object, T> FindBestMatch(ModalityType queryModality, object queryData, IEnumerable<ValueTuple<ModalityType, object>> candidates) Finds the best matching modality representation for a query.
ValueTuple<T, Dictionary<string, object>> ComputeAlignment(ModalityType modality1, object data1, ModalityType modality2, object data2) Computes the alignment between two modalities given paired data.
Vector<T> FuseModalities(Dictionary<ModalityType, Vector<T>> modalityEmbeddings, string fusionMethod) Performs multimodal fusion by combining embeddings from multiple modalities.
Vector<T> GetAudioEmbedding(Tensor<T> audioWaveform, int sampleRate) Converts audio into a shared embedding vector.
Vector<T> GetDepthEmbedding(Tensor<T> depthMap) Converts depth map into a shared embedding vector.
Vector<T> GetEmbedding(ModalityType modality, object data) Gets embedding for any supported modality using a generic interface.
Vector<T> GetIMUEmbedding(Tensor<T> imuData) Converts IMU sensor data into a shared embedding vector.
Vector<T> GetImageEmbedding(Tensor<T> image) Converts an image into a shared embedding vector.
Vector<T> GetTextEmbedding(string text) Converts text into a shared embedding vector.
Vector<T> GetThermalEmbedding(Tensor<T> thermalImage) Converts thermal image into a shared embedding vector.
Vector<T> GetVideoEmbedding(IEnumerable<Tensor<T>> frames) Converts video into a shared embedding vector.