LLaVAOneVision<T>

Models & Types · AiDotNet.VisionLanguage.InstructionTuned

LLaVA-OneVision: single model for images, multi-image, and videos with easy visual task transfer.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → LLaVAOneVision<T>

Implements: IInstructionTunedVLM<T>

For Beginners

LLaVA-OneVision is a unified model that handles single images, multiple images, and videos all in one architecture — previous models typically only handled one type. It uses the SigLIP-SO400M vision encoder (a shape-optimized version of SigLIP) with a Qwen2 language backbone. Through curriculum-based training — first learning from single images, then multi-image tasks, then video — the model develops strong visual understanding that transfers easily across different input types. This means you can use the same model for photo analysis, comparing multiple images, and understanding video content. Default values follow the original paper settings.

How It Works

LLaVA-OneVision (Li et al., 2024) extends LLaVA-NeXT to handle single images, multi-image, and video inputs in a single unified model. It uses SigLIP-SO400M vision encoder with Qwen2 language backbone and achieves strong performance across image, multi-image, and video benchmarks through curriculum-based visual instruction tuning.

References:

  • Paper: “LLaVA-OneVision: Easy Visual Task Transfer” (Li et al., 2024)

Example

// Create a LLaVA-OneVision model for unified image and video understanding
// using SigLIP-SO400M encoder with Qwen2 backbone
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new LLaVAOneVision<double>(architecture, "llavaonevision.onnx");
            
// Training mode with native layers
var trainModel = new LLaVAOneVision<double>(architecture, new LLaVAOneVisionOptions());

Constructors

Constructor Summary
LLaVAOneVision(NeuralNetworkArchitecture<T> architecture, LLaVAOneVisionOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
LLaVAOneVision(NeuralNetworkArchitecture<T> architecture, string modelPath, LLaVAOneVisionOptions options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }
string LanguageModelName { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage)
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates text using LLaVA-OneVision’s unified image/multi-image/video architecture.