MPLUGOwl3<T>
Models & Types · AiDotNet.VisionLanguage.InstructionTuned
mPLUG-Owl3: enhanced with hyper-attention for long visual sequences.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → MPLUGOwl3<T>
Implements: IInstructionTunedVLM<T>
For Beginners
mPLUG-Owl3 tackles one of the biggest challenges in vision-language models: efficiently processing long sequences of images or video frames. Standard attention has quadratic cost — processing twice as many visual tokens takes four times the compute. mPLUG-Owl3 introduces “hyper-attention” that handles long visual sequences much more efficiently, making it practical to process many images or long videos. It uses Qwen2 as the language backbone and an enhanced visual abstractor that can handle extended sequences without the usual performance bottleneck. Default values follow the original paper settings.
How It Works
mPLUG-Owl3 (Alibaba, 2024) introduces hyper-attention for efficiently processing long visual sequences. It uses Qwen2 as the language backbone with an enhanced visual abstractor that can handle extended visual token sequences without quadratic attention cost.
References:
- Paper: “mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models” (2024)
Example
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
var trainModel = new MPLUGOwl3<double>(architecture, new MPLUGOwl3Options());
Constructors
| Constructor | Summary |
|---|---|
MPLUGOwl3(NeuralNetworkArchitecture<T> architecture, MPLUGOwl3Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
MPLUGOwl3(NeuralNetworkArchitecture<T> architecture, string modelPath, MPLUGOwl3Options options) |
Properties
| Property | Summary |
|---|---|
int DecoderEmbeddingDim { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxGenerationLength { get; } |
|
string LanguageModelName { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage) |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) |
Generates text using mPLUG-Owl3’s hyper-attention architecture. |