Emu3<T>

Models & Types · AiDotNet.VisionLanguage.Generative

Emu3: next-token prediction unifies understanding and generation in a single model.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → Emu3<T>

Implements: IGenerativeVisionLanguageModel<T>

For Beginners

Emu3 simplifies multimodal AI by treating everything — text and images — as sequences of tokens predicted one at a time. Images are converted to discrete tokens via a VQVAE codebook, then mixed with text tokens in a unified vocabulary. A single autoregressive transformer handles both understanding and generation without needing separate diffusion or contrastive modules. Default values follow the original paper settings.

How It Works

Emu3 (Wang et al., 2024) simplifies the multimodal architecture by using next-token prediction as the sole training objective for both understanding and generation. Images are tokenized into discrete visual tokens via a VQVAE, then interleaved with text tokens in a unified vocabulary. A single autoregressive transformer generates both text and visual tokens.

References:

  • Paper: “Emu3: Next-Token Prediction is All You Need” (Wang et al., 2024)

Example

var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
var trainModel = new Emu3<double>(architecture, new Emu3Options());

Constructors

Constructor Summary
Emu3(NeuralNetworkArchitecture<T> architecture, Emu3Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
Emu3(NeuralNetworkArchitecture<T> architecture, string modelPath, Emu3Options options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates using Emu3’s next-token prediction architecture.