IGenerativeVisionLanguageModel<T>
Interfaces · AiDotNet.VisionLanguage.Interfaces
Interface for generative vision-language models that produce text output from visual input.
Implements: IVisualEncoder<T>
Known implementations: BLIP3<T>, CoCa<T>, Emu2<T>, Emu3<T>, Emu<T>, GIT<T>, IDEFICS2<T>, IDEFICS3<T>, IDEFICS<T>, IDocumentUnderstandingModel<T>, IInstructionTunedVLM<T>, IMedicalVLM<T>, IProprietaryVLM<T>, IRemoteSensingVLM<T>, IThreeDVisionLanguageModel<T>, IUnifiedVisionModel<T>, IVideoLanguageModel<T>, IVisionLanguageAction<T>, InstructBLIP<T>, KOSMOS1<T>, KOSMOS2<T>, OpenFlamingo<T>, PaLI3<T>, PaLIX<T>, PaLI<T>
How It Works
Generative VLMs take an image (and optionally a text prompt) and produce text output such as captions, answers to visual questions, or descriptions. Architectures include:
- Q-Former bridges (BLIP-2, InstructBLIP) - lightweight adapter between frozen encoders
- Encoder-decoder (GIT, CoCa, PaLI) - ViT encoder + autoregressive text decoder
- Perceiver resampler (Flamingo, IDEFICS) - latent queries cross-attend to vision features
- Causal multimodal (KOSMOS) - visual tokens embedded directly in causal LM
- Unified generation (Emu) - single model for understanding + generation
Properties
| Property | Summary |
|---|---|
int DecoderEmbeddingDim { get; } |
Gets the dimensionality of the decoder embedding space. |
int MaxGenerationLength { get; } |
Gets the maximum number of tokens the model can generate. |
Methods
| Method | Summary |
|---|---|
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) |
Generates output token logits from an image, optionally conditioned on a text prompt. |