QwenVL<T>

Models & Types · AiDotNet.VisionLanguage.InstructionTuned

Qwen-VL: visual window attention, multi-resolution, bounding box output via cross-attention resampler.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → QwenVL<T>

Implements: IInstructionTunedVLM<T>

For Beginners

Qwen-VL from Alibaba is a versatile model that goes beyond just describing images — it can locate objects by outputting bounding box coordinates. It uses visual window attention (processing image regions locally for efficiency) and a cross-attention resampler that compresses the large number of visual tokens into a fixed smaller set before feeding them to the language model. It supports images at multiple resolutions and can handle tasks like visual question answering, text reading (OCR), and visual grounding (finding where objects are in an image). Default values follow the original paper settings.

How It Works

Qwen-VL (Bai et al., 2023) uses a ViT vision encoder with visual window attention and a cross-attention resampler to compress visual features before feeding into the Qwen language model. It supports multi-resolution input and can output bounding box coordinates for visual grounding tasks.

References:

  • Paper: “Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond” (Bai et al., 2023)

Example

var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
var trainModel = new QwenVL<double>(architecture, new QwenVLOptions());

Constructors

Constructor Summary
QwenVL(NeuralNetworkArchitecture<T> architecture, QwenVLOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
QwenVL(NeuralNetworkArchitecture<T> architecture, string modelPath, QwenVLOptions options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }
string LanguageModelName { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage)
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates text using Qwen-VL’s window-attention + cross-attention resampler architecture.