LLaVA15<T>

Models & Types · AiDotNet.VisionLanguage.InstructionTuned

LLaVA-1.5: improved baselines with visual instruction tuning using MLP cross-modal connector.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → LLaVA15<T>

Implements: IInstructionTunedVLM<T>

For Beginners

LLaVA-1.5 is an improved version of the original LLaVA (Large Language and Vision Assistant) that makes simple but effective upgrades: it replaces the single linear projection with a two-layer MLP connector (for better visual-to-text mapping), uses a higher-resolution CLIP-ViT-L/14 encoder at 336 pixels (for sharper visual detail), and adds academic VQA training data. Despite these minimal changes, it achieved state-of-the-art results on 11 benchmarks, demonstrating that careful engineering of the basics often matters more than complex architectural innovations. Default values follow the original paper settings.

How It Works

LLaVA-1.5 (Liu et al., 2024) improves upon LLaVA by replacing the linear projection with a two-layer MLP cross-modal connector, using higher-resolution CLIP-ViT-L/14 at 336px, and adding academic-task-oriented VQA data. It achieves SOTA results on 11 benchmarks with simple modifications to the original LLaVA architecture.

References:

  • Paper: “Improved Baselines with Visual Instruction Tuning” (Liu et al., 2024)

Example

// Create a LLaVA-1.5 model for visual instruction following
// using CLIP-ViT-L/14 at 336px with MLP cross-modal connector
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new LLaVA15<double>(architecture, "llava15.onnx");
            
// Training mode with native layers
var trainModel = new LLaVA15<double>(architecture, new LLaVA15Options());

Constructors

Constructor Summary
LLaVA15(NeuralNetworkArchitecture<T> architecture, LLaVA15Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
LLaVA15(NeuralNetworkArchitecture<T> architecture, string modelPath, LLaVA15Options options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }
string LanguageModelName { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage)
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates text using LLaVA-1.5’s MLP cross-modal connector architecture.