LLaVA15<T>
Models & Types · AiDotNet.VisionLanguage.InstructionTuned
LLaVA-1.5: improved baselines with visual instruction tuning using MLP cross-modal connector.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → LLaVA15<T>
Implements: IInstructionTunedVLM<T>
For Beginners
LLaVA-1.5 is an improved version of the original LLaVA (Large Language and Vision Assistant) that makes simple but effective upgrades: it replaces the single linear projection with a two-layer MLP connector (for better visual-to-text mapping), uses a higher-resolution CLIP-ViT-L/14 encoder at 336 pixels (for sharper visual detail), and adds academic VQA training data. Despite these minimal changes, it achieved state-of-the-art results on 11 benchmarks, demonstrating that careful engineering of the basics often matters more than complex architectural innovations. Default values follow the original paper settings.
How It Works
LLaVA-1.5 (Liu et al., 2024) improves upon LLaVA by replacing the linear projection with a two-layer MLP cross-modal connector, using higher-resolution CLIP-ViT-L/14 at 336px, and adding academic-task-oriented VQA data. It achieves SOTA results on 11 benchmarks with simple modifications to the original LLaVA architecture.
References:
- Paper: “Improved Baselines with Visual Instruction Tuning” (Liu et al., 2024)
Example
// Create a LLaVA-1.5 model for visual instruction following
// using CLIP-ViT-L/14 at 336px with MLP cross-modal connector
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
// ONNX inference mode with pre-trained model
var model = new LLaVA15<double>(architecture, "llava15.onnx");
// Training mode with native layers
var trainModel = new LLaVA15<double>(architecture, new LLaVA15Options());
Constructors
| Constructor | Summary |
|---|---|
LLaVA15(NeuralNetworkArchitecture<T> architecture, LLaVA15Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
LLaVA15(NeuralNetworkArchitecture<T> architecture, string modelPath, LLaVA15Options options) |
Properties
| Property | Summary |
|---|---|
int DecoderEmbeddingDim { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxGenerationLength { get; } |
|
string LanguageModelName { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage) |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) |
Generates text using LLaVA-1.5’s MLP cross-modal connector architecture. |