MPLUGOwl<T>

Models & Types · AiDotNet.VisionLanguage.InstructionTuned

mPLUG-Owl: modular VLM with visual abstractor for vision-language alignment.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → MPLUGOwl<T>

Implements: IInstructionTunedVLM<T>

For Beginners

mPLUG-Owl from Alibaba uses a modular design where different components can be trained independently. Its key innovation is the “visual abstractor” — a module that sits between the vision encoder and language model, using learnable queries with cross-attention to compress and align visual features into a format the language model can understand. Think of it as a translator that converts raw image features into something the text model can work with effectively. The modular design means you can upgrade individual components without retraining the whole model. Default values follow the original paper settings.

How It Works

mPLUG-Owl (Alibaba, 2023) uses a modular architecture with a visual abstractor module that learns to compress and align visual features from the ViT encoder before feeding them into the LLaMA language model. The abstractor uses learnable queries with cross-attention.

References:

  • Paper: “mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality” (Ye et al., 2023)

Example

var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
var trainModel = new MPLUGOwl<double>(architecture, new MPLUGOwlOptions());

Constructors

Constructor Summary
MPLUGOwl(NeuralNetworkArchitecture<T> architecture, MPLUGOwlOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
MPLUGOwl(NeuralNetworkArchitecture<T> architecture, string modelPath, MPLUGOwlOptions options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }
string LanguageModelName { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage)
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates text using mPLUG-Owl’s modular architecture.