MPLUGOwl3<T>

Models & Types · AiDotNet.VisionLanguage.InstructionTuned

mPLUG-Owl3: enhanced with hyper-attention for long visual sequences.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → MPLUGOwl3<T>

Implements: IInstructionTunedVLM<T>

For Beginners

mPLUG-Owl3 tackles one of the biggest challenges in vision-language models: efficiently processing long sequences of images or video frames. Standard attention has quadratic cost — processing twice as many visual tokens takes four times the compute. mPLUG-Owl3 introduces “hyper-attention” that handles long visual sequences much more efficiently, making it practical to process many images or long videos. It uses Qwen2 as the language backbone and an enhanced visual abstractor that can handle extended sequences without the usual performance bottleneck. Default values follow the original paper settings.

How It Works

mPLUG-Owl3 (Alibaba, 2024) introduces hyper-attention for efficiently processing long visual sequences. It uses Qwen2 as the language backbone with an enhanced visual abstractor that can handle extended visual token sequences without quadratic attention cost.

References:

  • Paper: “mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models” (2024)

Example

var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
var trainModel = new MPLUGOwl3<double>(architecture, new MPLUGOwl3Options());

Constructors

Constructor Summary
MPLUGOwl3(NeuralNetworkArchitecture<T> architecture, MPLUGOwl3Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
MPLUGOwl3(NeuralNetworkArchitecture<T> architecture, string modelPath, MPLUGOwl3Options options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }
string LanguageModelName { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage)
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates text using mPLUG-Owl3’s hyper-attention architecture.