Molmo<T>

Models & Types · AiDotNet.VisionLanguage.InstructionTuned

Molmo: open VLM with pointing and grounding capabilities.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → Molmo<T>

Implements: IInstructionTunedVLM<T>

For Beginners

Molmo from AI2 is notable for being fully open — not just the model weights, but also the training data (called PixMo) is publicly available, which is rare for high-performing multimodal models. Beyond standard visual QA, Molmo can “point” to objects in images by outputting coordinate positions, making it useful for tasks like “where is the cat in this image?” where it can indicate the exact location. It achieves state-of-the-art performance among open models while being completely transparent in how it was trained. Default values follow the original paper settings.

How It Works

Molmo (AI2, 2024) from the Allen Institute for AI is a fully open multimodal model with both open weights and open training data (PixMo). It features pointing and grounding capabilities — the model can point to specific objects in images by generating (x, y) coordinates and can localize objects described in natural language. Molmo achieves state-of-the-art performance among open models while providing complete transparency in its training data and methodology.

References:

  • Paper: “Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models” (AI2, 2024)

Example

// Create a Molmo model for visual QA with pointing and grounding
// using fully open weights and training data
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new Molmo<double>(architecture, "molmo.onnx");
            
// Training mode with native layers
var trainModel = new Molmo<double>(architecture, new MolmoOptions());

Constructors

Constructor Summary
Molmo(NeuralNetworkArchitecture<T> architecture, MolmoOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
Molmo(NeuralNetworkArchitecture<T> architecture, string modelPath, MolmoOptions options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }
string LanguageModelName { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> Chat(Tensor<T> image, IEnumerable<ValueTuple<string, string>> conversationHistory, string userMessage)
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates text using Molmo’s pointing-capable architecture with diverse data mixture.