MPLUGOwl<T>
Models & Types · AiDotNet.VisionLanguage.InstructionTuned
mPLUG-Owl: modular VLM with visual abstractor for vision-language alignment.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → MPLUGOwl<T>
Implements: IInstructionTunedVLM<T>
For Beginners
mPLUG-Owl from Alibaba uses a modular design where different components can be trained independently. Its key innovation is the “visual abstractor” — a module that sits between the vision encoder and language model, using learnable queries with cross-attention to compress and align visual features into a format the language model can understand. Think of it as a translator that converts raw image features into something the text model can work with effectively. The modular design means you can upgrade individual components without retraining the whole model. Default values follow the original paper settings.
How It Works
mPLUG-Owl (Alibaba, 2023) uses a modular architecture with a visual abstractor module that learns to compress and align visual features from the ViT encoder before feeding them into the LLaMA language model. The abstractor uses learnable queries with cross-attention.
References:
- Paper: “mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality” (Ye et al., 2023)
Example
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
var trainModel = new MPLUGOwl<double>(architecture, new MPLUGOwlOptions());
Constructors
| Constructor | Summary |
|---|---|
MPLUGOwl(NeuralNetworkArchitecture<T> architecture, MPLUGOwlOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
MPLUGOwl(NeuralNetworkArchitecture<T> architecture, string modelPath, MPLUGOwlOptions options) |
Properties
| Property | Summary |
|---|---|
int DecoderEmbeddingDim { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxGenerationLength { get; } |
|
string LanguageModelName { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage) |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) |
Generates text using mPLUG-Owl’s modular architecture. |