MPLUGOwl2<T>

Models & Types · AiDotNet.VisionLanguage.InstructionTuned

mPLUG-Owl2: improved modular design for multi-image understanding.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → MPLUGOwl2<T>

Implements: IInstructionTunedVLM<T>

For Beginners

mPLUG-Owl2 improves on its predecessor with an enhanced visual abstractor and LLaMA-2 backbone for better multi-image understanding. The key upgrade is “modality collaboration” — the model learns to better coordinate between different types of input (images, text, and their relationships) rather than treating them independently. This gives it improved reasoning capabilities, especially when dealing with multiple images or complex visual scenes that require understanding spatial relationships and interactions between objects. Default values follow the original paper settings.

How It Works

mPLUG-Owl2 (Alibaba, 2024) improves upon mPLUG-Owl with an enhanced visual abstractor module and LLaMA-2 backbone for better multi-image understanding and reasoning capabilities.

References:

  • Paper: “mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration” (2024)

Example

var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
var trainModel = new MPLUGOwl2<double>(architecture, new MPLUGOwl2Options());

Constructors

Constructor Summary
MPLUGOwl2(NeuralNetworkArchitecture<T> architecture, MPLUGOwl2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
MPLUGOwl2(NeuralNetworkArchitecture<T> architecture, string modelPath, MPLUGOwl2Options options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }
string LanguageModelName { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage)
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates text using mPLUG-Owl2’s enhanced modular architecture.