Pixtral<T>

Models & Types · AiDotNet.VisionLanguage.InstructionTuned

Pixtral: Mistral’s 12B decoder + 400M vision encoder.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → Pixtral<T>

Implements: IInstructionTunedVLM<T>

For Beginners

Pixtral is Mistral’s entry into multimodal AI, pairing a compact 400M parameter vision encoder with the powerful 12B Mistral language model. It processes high-resolution images at 1024 pixels and uses an MLP projection to connect visual features to the language decoder. The result is a model that combines Mistral’s strong language capabilities (reasoning, coding, instruction following) with image understanding. It can describe images, answer visual questions, read text in photos, and reason about visual content. Default values follow the original paper settings.

How It Works

Pixtral (Mistral, 2024) is a 12B parameter multimodal model combining a 400M parameter vision encoder with the Mistral language model. It processes high-resolution images at 1024px and uses MLP projection for vision-language alignment.

References:

  • Paper: “Pixtral 12B” (Mistral, 2024)

Example

// Create a Pixtral model for high-resolution visual understanding
// using 400M vision encoder with Mistral 12B backbone
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new Pixtral<double>(architecture, "pixtral.onnx");
            
// Training mode with native layers
var trainModel = new Pixtral<double>(architecture, new PixtralOptions());

Constructors

Constructor Summary
Pixtral(NeuralNetworkArchitecture<T> architecture, PixtralOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
Pixtral(NeuralNetworkArchitecture<T> architecture, string modelPath, PixtralOptions options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }
string LanguageModelName { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage)
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates text using Pixtral’s jointly-trained vision encoder architecture.