LLaVANeXT<T>
Models & Types · AiDotNet.VisionLanguage.InstructionTuned
LLaVA-NeXT: improved reasoning, OCR, and world knowledge via dynamic high-resolution AnyRes.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → LLaVANeXT<T>
Implements: IInstructionTunedVLM<T>
For Beginners
LLaVA-NeXT extends LLaVA-1.5 with a key innovation called AnyRes (Any Resolution) — instead of resizing all images to a fixed size, it splits high-resolution images into multiple tiles and processes each tile separately. This dramatically improves the model’s ability to read text in images (OCR), understand charts and tables, and reason about fine visual details. It uses LLaMA-3 as the language backbone for stronger reasoning. The result is a model much better at tasks requiring visual precision, like reading small text or understanding complex diagrams. Default values follow the original paper settings.
How It Works
LLaVA-NeXT (Liu et al., 2024) extends LLaVA-1.5 with AnyRes dynamic resolution that processes high-resolution images by splitting them into multiple tiles, significantly improving OCR, chart understanding, and fine-grained visual reasoning while using LLaMA-3 as the language backbone.
References:
- Paper: “LLaVA-NeXT: Improved reasoning, OCR, and world knowledge” (Liu et al., 2024)
Example
// Create a LLaVA-NeXT model for high-resolution visual reasoning
// using AnyRes dynamic resolution with LLaMA-3 backbone
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
// ONNX inference mode with pre-trained model
var model = new LLaVANeXT<double>(architecture, "llavanext.onnx");
// Training mode with native layers
var trainModel = new LLaVANeXT<double>(architecture, new LLaVANeXTOptions());
Constructors
| Constructor | Summary |
|---|---|
LLaVANeXT(NeuralNetworkArchitecture<T> architecture, LLaVANeXTOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
LLaVANeXT(NeuralNetworkArchitecture<T> architecture, string modelPath, LLaVANeXTOptions options) |
Properties
| Property | Summary |
|---|---|
int DecoderEmbeddingDim { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxGenerationLength { get; } |
|
string LanguageModelName { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage) |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) |
Generates text using LLaVA-NeXT’s AnyRes dynamic high-resolution architecture. |