Emu3<T>
Models & Types · AiDotNet.VisionLanguage.Generative
Emu3: next-token prediction unifies understanding and generation in a single model.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → Emu3<T>
Implements: IGenerativeVisionLanguageModel<T>
For Beginners
Emu3 simplifies multimodal AI by treating everything — text and images — as sequences of tokens predicted one at a time. Images are converted to discrete tokens via a VQVAE codebook, then mixed with text tokens in a unified vocabulary. A single autoregressive transformer handles both understanding and generation without needing separate diffusion or contrastive modules. Default values follow the original paper settings.
How It Works
Emu3 (Wang et al., 2024) simplifies the multimodal architecture by using next-token prediction as the sole training objective for both understanding and generation. Images are tokenized into discrete visual tokens via a VQVAE, then interleaved with text tokens in a unified vocabulary. A single autoregressive transformer generates both text and visual tokens.
References:
- Paper: “Emu3: Next-Token Prediction is All You Need” (Wang et al., 2024)
Example
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
var trainModel = new Emu3<double>(architecture, new Emu3Options());
Constructors
| Constructor | Summary |
|---|---|
Emu3(NeuralNetworkArchitecture<T> architecture, Emu3Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
Emu3(NeuralNetworkArchitecture<T> architecture, string modelPath, Emu3Options options) |
Properties
| Property | Summary |
|---|---|
int DecoderEmbeddingDim { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxGenerationLength { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) |
Generates using Emu3’s next-token prediction architecture. |