Emu<T>
Models & Types · AiDotNet.VisionLanguage.Generative
Emu: unified VQA, captioning, and image generation via EVA-CLIP + LLM + regression head.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → Emu<T>
Implements: IGenerativeVisionLanguageModel<T>
For Beginners
Emu unifies visual understanding (answering questions about images, generating captions) and image generation in a single model. It uses an EVA-CLIP vision encoder with a LLaMA-based language model, and a visual regression head that can map LLM outputs back to visual embeddings for image generation via diffusion. Default values follow the original paper settings.
How It Works
Emu (Sun et al., 2023) unifies visual understanding and generation in a single model. It uses an EVA-CLIP vision encoder to extract visual features, a Causal Transformer (LLaMA-based) as the multimodal LLM, and a visual regression head that maps LLM hidden states back to the visual embedding space for image generation via a diffusion decoder.
References:
- Paper: “Generative Pretraining in Multimodality” (Sun et al., 2023)
Example
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
var trainModel = new Emu<double>(architecture, new EmuOptions());
Constructors
| Constructor | Summary |
|---|---|
Emu(NeuralNetworkArchitecture<T> architecture, EmuOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
Emu(NeuralNetworkArchitecture<T> architecture, string modelPath, EmuOptions options) |
Properties
| Property | Summary |
|---|---|
int DecoderEmbeddingDim { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxGenerationLength { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) |
Generates using Emu’s unified understanding + generation architecture. |