LLaVAOneVision<T>
Models & Types · AiDotNet.VisionLanguage.InstructionTuned
LLaVA-OneVision: single model for images, multi-image, and videos with easy visual task transfer.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → LLaVAOneVision<T>
Implements: IInstructionTunedVLM<T>
For Beginners
LLaVA-OneVision is a unified model that handles single images, multiple images, and videos all in one architecture — previous models typically only handled one type. It uses the SigLIP-SO400M vision encoder (a shape-optimized version of SigLIP) with a Qwen2 language backbone. Through curriculum-based training — first learning from single images, then multi-image tasks, then video — the model develops strong visual understanding that transfers easily across different input types. This means you can use the same model for photo analysis, comparing multiple images, and understanding video content. Default values follow the original paper settings.
How It Works
LLaVA-OneVision (Li et al., 2024) extends LLaVA-NeXT to handle single images, multi-image, and video inputs in a single unified model. It uses SigLIP-SO400M vision encoder with Qwen2 language backbone and achieves strong performance across image, multi-image, and video benchmarks through curriculum-based visual instruction tuning.
References:
- Paper: “LLaVA-OneVision: Easy Visual Task Transfer” (Li et al., 2024)
Example
// Create a LLaVA-OneVision model for unified image and video understanding
// using SigLIP-SO400M encoder with Qwen2 backbone
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
// ONNX inference mode with pre-trained model
var model = new LLaVAOneVision<double>(architecture, "llavaonevision.onnx");
// Training mode with native layers
var trainModel = new LLaVAOneVision<double>(architecture, new LLaVAOneVisionOptions());
Constructors
| Constructor | Summary |
|---|---|
LLaVAOneVision(NeuralNetworkArchitecture<T> architecture, LLaVAOneVisionOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
LLaVAOneVision(NeuralNetworkArchitecture<T> architecture, string modelPath, LLaVAOneVisionOptions options) |
Properties
| Property | Summary |
|---|---|
int DecoderEmbeddingDim { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxGenerationLength { get; } |
|
string LanguageModelName { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage) |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) |
Generates text using LLaVA-OneVision’s unified image/multi-image/video architecture. |