IVisionLanguageFusionModel<T>

Interfaces · AiDotNet.VisionLanguage.Interfaces

Interface for vision-language fusion models that combine image and text features for tasks like VQA, image-text matching, and cross-modal retrieval.

Implements: IVisualEncoder<T>

Known implementations: BridgeTower<T>, LXMERT<T>, METER<T>, Oscar<T>, UNITER<T>, ViLBERT<T>, ViLT<T>, VinVL<T>, VisualBERT<T>

How It Works

Unlike contrastive models that keep vision and text separate, fusion models combine modalities internally via co-attention, cross-attention, or single-stream architectures. They support tasks including:

  • Visual Question Answering (VQA) - answering questions about images
  • Image-Text Matching (ITM) - determining if image and text match
  • Cross-Modal Retrieval - finding relevant images for text or vice versa
  • Visual Entailment - reasoning about image-text relationships

Properties

Property Summary
int FusionEmbeddingDim { get; } Gets the fusion embedding dimension.
int MaxSequenceLength { get; } Gets the maximum text sequence length.

Methods

Method Summary
T ComputeMatchingScore(Tensor<T> image, string text) Computes an image-text matching score indicating how well the pair matches.
Tensor<T> FuseImageText(Tensor<T> image, string text) Computes a fused representation of an image-text pair.