ImageBindNeuralNetwork<T>
Models & Types · AiDotNet.NeuralNetworks
ImageBind neural network for binding multiple modalities (6+) into a shared embedding space.
Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → MultimodalModelLayoutBase<T> → ImageBindNeuralNetwork<T>
Implements: IImageBindModel<T>
For Beginners
ImageBind connects ALL types of data together!
Architecture overview:
- Modality-Specific Encoders: Each modality has its own encoder (ViT for images, Transformer for text, etc.)
- Projection Heads: Map each modality’s features to the shared embedding space
- Contrastive Learning: Align modalities using image as the bridge modality
Key capabilities:
- Cross-modal retrieval: Find images matching audio, text matching video, etc.
- Zero-shot classification: Classify any modality using text labels
- Emergent alignment: Compare modalities never directly paired during training
How It Works
ImageBind learns a joint embedding space across multiple modalities: images, text, audio, depth, thermal, and IMU data. It uses images as a binding modality - since web data contains many (image, text) pairs, (image, audio) pairs from videos, etc., the model can learn cross-modal relationships even without direct pairs between all modalities.
Example
var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var image = Tensor<float>.CreateRandom(new[] { 1, 3, 224, 224 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new ImageBindNeuralNetwork<float>(architecture))
.Build(trainX, trainY);
var embedding = result.Predict(image);
Constructors
| Constructor | Summary |
|---|---|
ImageBindNeuralNetwork(NeuralNetworkArchitecture<T> architecture, ImageBindOptions options, ITokenizer tokenizer, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
Creates an ImageBind network using native library layers. |
ImageBindNeuralNetwork(NeuralNetworkArchitecture<T> architecture, string imageEncoderPath, string textEncoderPath, string audioEncoderPath, ITokenizer tokenizer, ImageBindOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
Creates an ImageBind network using pretrained ONNX models. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<ModalityType> SupportedModalities { get; } |
|
int EmbeddingDimension { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> ZeroShotClassify(ModalityType modality, object data, IEnumerable<string> classLabels) |
|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
|
IEnumerable<ValueTuple<int, T>> CrossModalRetrieval(Vector<T> queryEmbedding, IEnumerable<Vector<T>> targetEmbeddings, int topK) |
|
IEnumerable<ValueTuple<string, T>> GenerateDescriptions(ModalityType modality, object data, IEnumerable<string> candidateDescriptions, int topK) |
|
ModelMetadata<T> GetModelMetadata() |
|
T ComputeCrossModalSimilarity(Vector<T> embedding1, Vector<T> embedding2) |
|
T ComputeEmergentAudioTextSimilarity(Tensor<T> audio, string text) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
ValueTuple<ModalityType, object, T> FindBestMatch(ModalityType queryModality, object queryData, IEnumerable<ValueTuple<ModalityType, object>> candidates) |
|
ValueTuple<T, Dictionary<string, object>> ComputeAlignment(ModalityType modality1, object data1, ModalityType modality2, object data2) |
|
Vector<T> FuseModalities(Dictionary<ModalityType, Vector<T>> modalityEmbeddings, string fusionMethod) |
|
Vector<T> GetAudioEmbedding(Tensor<T> audioWaveform, int sampleRate) |
|
Vector<T> GetDepthEmbedding(Tensor<T> depthMap) |
|
Vector<T> GetEmbedding(ModalityType modality, object data) |
|
Vector<T> GetIMUEmbedding(Tensor<T> imuData) |
|
Vector<T> GetImageEmbedding(Tensor<T> image) |
|
Vector<T> GetTextEmbedding(string text) |
|
Vector<T> GetThermalEmbedding(Tensor<T> thermalImage) |
|
Vector<T> GetVideoEmbedding(IEnumerable<Tensor<T>> frames) |