AquilaVL<T>

Models & Types · AiDotNet.VisionLanguage.InstructionTuned

AquilaVL: bilingual VLM aligned with Chinese and English visual-language understanding.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → AquilaVL<T>

Implements: IInstructionTunedVLM<T>

For Beginners

AquilaVL is a bilingual (Chinese/English) model that can look at an image and answer questions about it in either language. It uses a SigLIP vision encoder to understand what is in the image, projects those visual features through an MLP into the Qwen2.5 language model, and generates text responses. It handles images at their native resolution by splitting them into tiles, so it can understand both the big picture and fine details. Default values follow the original paper settings.

How It Works

AquilaVL (BAAI, 2024) is a bilingual vision-language model designed for strong performance in both Chinese and English visual understanding. It follows a LLaVA-style architecture using a SigLIP vision encoder to extract dense visual features, a 2-layer MLP projector with SiLU activation to map visual tokens into the language model’s embedding space, and a Qwen2.5 decoder backbone for text generation. AquilaVL supports dynamic resolution by tiling images and encoding each tile independently, enabling high-fidelity understanding of detailed scenes.

References:

  • Paper: “AquilaVL: Advanced Vision-Language Model” (BAAI, 2024)

Example

// Create an AquilaVL model for bilingual Chinese/English visual QA
// using SigLIP encoder + MLP projector + Qwen2.5 decoder
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new AquilaVL<double>(architecture, "aquilavl.onnx");
            
// Training mode with native layers
var trainModel = new AquilaVL<double>(architecture, new AquilaVLOptions());

Constructors

Constructor Summary
AquilaVL(NeuralNetworkArchitecture<T> architecture, AquilaVLOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
AquilaVL(NeuralNetworkArchitecture<T> architecture, string modelPath, AquilaVLOptions options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }
string LanguageModelName { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage)
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates text using Aquila-VL’s SigLIP + Qwen2.5 architecture.