Chameleon<T>

Models & Types · AiDotNet.VisionLanguage.Unified

Chameleon: early fusion with discrete tokens for all modalities.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → Chameleon<T>

Implements: IUnifiedVisionModel<T>

For Beginners

Chameleon is a unified model that handles text and images using the same discrete token vocabulary. Default values follow the original paper settings.

How It Works

Chameleon (Meta, 2024) is a mixed-modal early-fusion foundation model that represents all modalities (text, images, code) as discrete tokens in a unified vocabulary. Images are quantized via a VQ-VAE encoder into discrete visual tokens, enabling a single autoregressive transformer to generate and understand any combination of text and images natively.

References:

  • Paper: “Chameleon: Mixed-Modal Early-Fusion Foundation Models” (Meta, 2024)

Example

// Create a Chameleon model for mixed-modal early-fusion generation
// with discrete tokens for all modalities in a unified vocabulary
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
            
// ONNX inference mode with pre-trained model
var model = new Chameleon<double>(architecture, "chameleon.onnx");
            
// Training mode with native layers
var trainModel = new Chameleon<double>(architecture, new ChameleonOptions());

Constructors

Constructor Summary
Chameleon(NeuralNetworkArchitecture<T> architecture, ChameleonOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
Chameleon(NeuralNetworkArchitecture<T> architecture, string modelPath, ChameleonOptions options)

Properties

Property Summary
bool SupportsGeneration { get; }
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates text from an image using Chameleon’s early-fusion mixed-modal approach.
Tensor<T> GenerateImage(string textDescription) Generates an image from text using Chameleon’s early-fusion discrete token generation.