MPLUGOwl2<T>
Models & Types · AiDotNet.VisionLanguage.InstructionTuned
mPLUG-Owl2: improved modular design for multi-image understanding.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → MPLUGOwl2<T>
Implements: IInstructionTunedVLM<T>
For Beginners
mPLUG-Owl2 improves on its predecessor with an enhanced visual abstractor and LLaMA-2 backbone for better multi-image understanding. The key upgrade is “modality collaboration” — the model learns to better coordinate between different types of input (images, text, and their relationships) rather than treating them independently. This gives it improved reasoning capabilities, especially when dealing with multiple images or complex visual scenes that require understanding spatial relationships and interactions between objects. Default values follow the original paper settings.
How It Works
mPLUG-Owl2 (Alibaba, 2024) improves upon mPLUG-Owl with an enhanced visual abstractor module and LLaMA-2 backbone for better multi-image understanding and reasoning capabilities.
References:
- Paper: “mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration” (2024)
Example
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
var trainModel = new MPLUGOwl2<double>(architecture, new MPLUGOwl2Options());
Constructors
| Constructor | Summary |
|---|---|
MPLUGOwl2(NeuralNetworkArchitecture<T> architecture, MPLUGOwl2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
MPLUGOwl2(NeuralNetworkArchitecture<T> architecture, string modelPath, MPLUGOwl2Options options) |
Properties
| Property | Summary |
|---|---|
int DecoderEmbeddingDim { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxGenerationLength { get; } |
|
string LanguageModelName { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage) |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) |
Generates text using mPLUG-Owl2’s enhanced modular architecture. |