Maya<T>

Models & Types · AiDotNet.VisionLanguage.InstructionTuned

Maya: multilingual multimodal VLM supporting 8 languages.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → Maya<T>

Implements: IInstructionTunedVLM<T>

For Beginners

Maya is built specifically for multilingual use — it can look at an image and answer questions about it in 8 different languages. While most vision-language models work well only in English, Maya was instruction-tuned with visual QA data across multiple languages so it can understand and respond in Hindi, Spanish, French, Arabic, Bengali, Chinese, Japanese, and English. This makes it useful for applications serving diverse global audiences. Default values follow the original paper settings.

How It Works

Maya (Gupta et al., 2024) is a multilingual multimodal model designed to understand images and generate text in 8 languages. Unlike most VLMs that focus primarily on English, Maya uses instruction fine-tuning with multilingual visual question answering data to support diverse languages including Hindi, Spanish, French, Arabic, Bengali, Chinese, Japanese, and English. It follows the LLaVA-style architecture with a vision encoder and language model connected through a projector.

References:

  • Paper: “Maya: An Instruction Finetuned Multilingual Multimodal Model” (2024)

Example

// Create a Maya model for multilingual visual question answering
// supporting 8 languages with instruction fine-tuning
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new Maya<double>(architecture, "maya.onnx");
            
// Training mode with native layers
var trainModel = new Maya<double>(architecture, new MayaOptions());

Constructors

Constructor Summary
Maya(NeuralNetworkArchitecture<T> architecture, MayaOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
Maya(NeuralNetworkArchitecture<T> architecture, string modelPath, MayaOptions options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }
string LanguageModelName { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage)
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates text using Maya’s multilingual vision-language architecture.