LLaVANeXT<T>

Models & Types · AiDotNet.VisionLanguage.InstructionTuned

LLaVA-NeXT: improved reasoning, OCR, and world knowledge via dynamic high-resolution AnyRes.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → LLaVANeXT<T>

Implements: IInstructionTunedVLM<T>

For Beginners

LLaVA-NeXT extends LLaVA-1.5 with a key innovation called AnyRes (Any Resolution) — instead of resizing all images to a fixed size, it splits high-resolution images into multiple tiles and processes each tile separately. This dramatically improves the model’s ability to read text in images (OCR), understand charts and tables, and reason about fine visual details. It uses LLaMA-3 as the language backbone for stronger reasoning. The result is a model much better at tasks requiring visual precision, like reading small text or understanding complex diagrams. Default values follow the original paper settings.

How It Works

LLaVA-NeXT (Liu et al., 2024) extends LLaVA-1.5 with AnyRes dynamic resolution that processes high-resolution images by splitting them into multiple tiles, significantly improving OCR, chart understanding, and fine-grained visual reasoning while using LLaMA-3 as the language backbone.

References:

  • Paper: “LLaVA-NeXT: Improved reasoning, OCR, and world knowledge” (Liu et al., 2024)

Example

// Create a LLaVA-NeXT model for high-resolution visual reasoning
// using AnyRes dynamic resolution with LLaMA-3 backbone
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new LLaVANeXT<double>(architecture, "llavanext.onnx");
            
// Training mode with native layers
var trainModel = new LLaVANeXT<double>(architecture, new LLaVANeXTOptions());

Constructors

Constructor Summary
LLaVANeXT(NeuralNetworkArchitecture<T> architecture, LLaVANeXTOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
LLaVANeXT(NeuralNetworkArchitecture<T> architecture, string modelPath, LLaVANeXTOptions options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }
string LanguageModelName { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage)
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates text using LLaVA-NeXT’s AnyRes dynamic high-resolution architecture.