Mantis<T>

Models & Types · AiDotNet.VisionLanguage.InstructionTuned

Mantis: interleaved multi-image VLM for complex visual reasoning.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → Mantis<T>

Implements: IInstructionTunedVLM<T>

For Beginners

Most vision-language models process one image at a time, but Mantis is designed to handle multiple images together. It can compare two photos, track changes across a sequence of images, or reason about relationships between different views. It does this through interleaved instruction tuning — training on examples where multiple images and text are woven together, teaching the model to understand how images relate to each other. This makes it great for tasks like “what changed between these two photos?” or “describe the story across these images.” Default values follow the original paper settings.

How It Works

Mantis (Jiang et al., 2024) is specifically designed for multi-image reasoning tasks where the model must understand relationships between multiple images presented together. It uses interleaved multi-image instruction tuning, where images and text are interleaved in the input sequence, enabling the model to compare, contrast, and reason across multiple visual inputs simultaneously. This is particularly useful for tasks like “spot the difference”, visual storytelling across image sequences, and multi-view reasoning.

References:

  • Paper: “Mantis: Interleaved Multi-Image Instruction Tuning” (2024)

Example

// Create a Mantis model for multi-image visual reasoning
// using interleaved image-text instruction tuning
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new Mantis<double>(architecture, "mantis.onnx");
            
// Training mode with native layers
var trainModel = new Mantis<double>(architecture, new MantisOptions());

Constructors

Constructor Summary
Mantis(NeuralNetworkArchitecture<T> architecture, MantisOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
Mantis(NeuralNetworkArchitecture<T> architecture, string modelPath, MantisOptions options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }
string LanguageModelName { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage)
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates text using Mantis’s multi-image interleaved attention architecture.