MiniGPT4<T>
Models & Types · AiDotNet.VisionLanguage.InstructionTuned
MiniGPT-4: ViT + Q-Former aligned with Vicuna via single projection layer.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → MiniGPT4<T>
Implements: IInstructionTunedVLM<T>
For Beginners
MiniGPT-4 was one of the first models to show that you can connect a frozen image understanding system (ViT + Q-Former from BLIP-2) to a powerful language model (Vicuna) with just a single linear projection layer and get impressive results. Its two-stage training is simple: first learn on millions of image-text pairs to align visual and text features, then fine-tune on a small set of curated instructions. This minimal approach unlocked capabilities like detailed image descriptions, creative writing from images, and visual reasoning that surprised the research community. Default values follow the original paper settings.
How It Works
MiniGPT-4 (Zhu et al., 2023) aligns a frozen ViT + Q-Former visual encoder (from BLIP-2) with the Vicuna language model using a single linear projection layer. Two-stage training (pretrain on image-text pairs, then fine-tune on curated instruction data) enables emergent capabilities like detailed image description and creative writing from images.
References:
- Paper: “MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models” (Zhu et al., 2023)
Example
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
var trainModel = new MiniGPT4<double>(architecture, new MiniGPT4Options());
Constructors
| Constructor | Summary |
|---|---|
MiniGPT4(NeuralNetworkArchitecture<T> architecture, MiniGPT4Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
MiniGPT4(NeuralNetworkArchitecture<T> architecture, string modelPath, MiniGPT4Options options) |
Properties
| Property | Summary |
|---|---|
int DecoderEmbeddingDim { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxGenerationLength { get; } |
|
string LanguageModelName { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage) |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) |
Generates text using MiniGPT-4’s Q-Former + linear projection architecture. |