IInstructionTunedVLM<T>
Interfaces · AiDotNet.VisionLanguage.Interfaces
Interface for instruction-tuned vision-language models that generate conversational responses from visual input conditioned on user instructions.
Implements: IGenerativeVisionLanguageModel<T>
Known implementations: AquilaVL<T>, Aria<T>, Cambrian1<T>, CogVLM2<T>, CogVLM<T>, DeepSeekVL2<T>, DeepSeekVL<T>, Dragonfly<T>, Eagle25<T>, Eagle<T>, Fuyu<T>, Gemma3<T>, IReasoningVLM<T>, InternVL25<T>, InternVL2<T>, InternVL3<T>, InternVL<T>, LLaVA15<T>, LLaVANeXT<T>, LLaVAOneVision15<T>, LLaVAOneVision<T>, Llama32Vision<T>, MPLUGOwl2<T>, MPLUGOwl3<T>, MPLUGOwl<T>, Mantis<T>, Maya<T>, MiniCPMV<T>, MiniCPMo<T>, MiniGPT4<T>, MiniGPTv2<T>, Molmo<T>, Monkey<T>, Moondream<T>, NVLM<T>, Ovis<T>, Phi3Vision<T>, Phi4Multimodal<T>, PixtralLarge<T>, Pixtral<T>, Qwen25VL<T>, Qwen2VL<T>, Qwen3VL<T>, QwenVL<T>, SmolVLM<T>, VILAU<T>, VILA<T>
How It Works
Instruction-tuned VLMs extend generative VLMs with instruction-following capabilities. They are fine-tuned on visual instruction data to enable conversational AI about images. Architectures include:
- MLP projection (LLaVA, InternVL): ViT → MLP connector → LLM
- Q-Former projection (MiniGPT-4): ViT → Q-Former → linear → LLM
- Cross-attention resampler (Qwen-VL): ViT → resampler → LLM
- Visual expert (CogVLM): ViT → visual expert modules in every LLM layer
Properties
| Property | Summary |
|---|---|
string LanguageModelName { get; } |
Gets the name of the language model backbone (e.g., “LLaMA”, “Vicuna”, “Qwen2”, “InternLM2”). |
Methods
| Method | Summary |
|---|---|
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage) |
Generates a response in a multi-turn chat context with visual input. |