Monkey<T>

Models & Types · AiDotNet.VisionLanguage.InstructionTuned

Monkey: high-resolution VLM with multi-level description generation.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → Monkey<T>

Implements: IInstructionTunedVLM<T>

For Beginners

Monkey focuses on two things that turn out to be critical for vision-language models: high image resolution and high-quality text labels. Most models downscale images to a fixed small size, losing fine details. Monkey instead processes images at up to 1344x896 resolution by splitting them into patches. It also uses carefully curated training data with detailed, accurate text descriptions. The result is a model that can generate descriptions at multiple levels of detail — from a brief one-line caption to a rich multi-paragraph description — and excels at understanding fine visual details like small text and intricate patterns. Default values follow the original paper settings.

How It Works

Monkey (Li et al., 2024) demonstrates that image resolution and text label quality are crucial for multimodal model performance. It processes images at high resolution (up to 1344x896) by dividing them into patches and encoding each at full resolution, then generates multi-level descriptions — from brief captions to detailed paragraph-level descriptions — using a carefully curated training dataset with high-quality text annotations.

References:

  • Paper: “Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models” (2024)

Example

// Create a Monkey model for high-resolution visual understanding
// with multi-level description generation
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new Monkey<double>(architecture, "monkey.onnx");
            
// Training mode with native layers
var trainModel = new Monkey<double>(architecture, new MonkeyOptions());

Constructors

Constructor Summary
Monkey(NeuralNetworkArchitecture<T> architecture, MonkeyOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
Monkey(NeuralNetworkArchitecture<T> architecture, string modelPath, MonkeyOptions options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }
string LanguageModelName { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage)
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates text using Monkey’s multi-level high-resolution description architecture.