IGenerativeVisionLanguageModel<T>

Interfaces · AiDotNet.VisionLanguage.Interfaces

Interface for generative vision-language models that produce text output from visual input.

Implements: IVisualEncoder<T>

Known implementations: BLIP3<T>, CoCa<T>, Emu2<T>, Emu3<T>, Emu<T>, GIT<T>, IDEFICS2<T>, IDEFICS3<T>, IDEFICS<T>, IDocumentUnderstandingModel<T>, IInstructionTunedVLM<T>, IMedicalVLM<T>, IProprietaryVLM<T>, IRemoteSensingVLM<T>, IThreeDVisionLanguageModel<T>, IUnifiedVisionModel<T>, IVideoLanguageModel<T>, IVisionLanguageAction<T>, InstructBLIP<T>, KOSMOS1<T>, KOSMOS2<T>, OpenFlamingo<T>, PaLI3<T>, PaLIX<T>, PaLI<T>

How It Works

Generative VLMs take an image (and optionally a text prompt) and produce text output such as captions, answers to visual questions, or descriptions. Architectures include:

  • Q-Former bridges (BLIP-2, InstructBLIP) - lightweight adapter between frozen encoders
  • Encoder-decoder (GIT, CoCa, PaLI) - ViT encoder + autoregressive text decoder
  • Perceiver resampler (Flamingo, IDEFICS) - latent queries cross-attend to vision features
  • Causal multimodal (KOSMOS) - visual tokens embedded directly in causal LM
  • Unified generation (Emu) - single model for understanding + generation

Properties

Property Summary
int DecoderEmbeddingDim { get; } Gets the dimensionality of the decoder embedding space.
int MaxGenerationLength { get; } Gets the maximum number of tokens the model can generate.

Methods

Method Summary
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates output token logits from an image, optionally conditioned on a text prompt.