IVisionLanguageFusionModel<T>
Interfaces · AiDotNet.VisionLanguage.Interfaces
Interface for vision-language fusion models that combine image and text features for tasks like VQA, image-text matching, and cross-modal retrieval.
Implements: IVisualEncoder<T>
Known implementations: BridgeTower<T>, LXMERT<T>, METER<T>, Oscar<T>, UNITER<T>, ViLBERT<T>, ViLT<T>, VinVL<T>, VisualBERT<T>
How It Works
Unlike contrastive models that keep vision and text separate, fusion models combine modalities internally via co-attention, cross-attention, or single-stream architectures. They support tasks including:
- Visual Question Answering (VQA) - answering questions about images
- Image-Text Matching (ITM) - determining if image and text match
- Cross-Modal Retrieval - finding relevant images for text or vice versa
- Visual Entailment - reasoning about image-text relationships
Properties
| Property | Summary |
|---|---|
int FusionEmbeddingDim { get; } |
Gets the fusion embedding dimension. |
int MaxSequenceLength { get; } |
Gets the maximum text sequence length. |
Methods
| Method | Summary |
|---|---|
T ComputeMatchingScore(Tensor<T> image, string text) |
Computes an image-text matching score indicating how well the pair matches. |
Tensor<T> FuseImageText(Tensor<T> image, string text) |
Computes a fused representation of an image-text pair. |