Emu<T>

Models & Types · AiDotNet.VisionLanguage.Generative

Emu: unified VQA, captioning, and image generation via EVA-CLIP + LLM + regression head.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → Emu<T>

Implements: IGenerativeVisionLanguageModel<T>

For Beginners

Emu unifies visual understanding (answering questions about images, generating captions) and image generation in a single model. It uses an EVA-CLIP vision encoder with a LLaMA-based language model, and a visual regression head that can map LLM outputs back to visual embeddings for image generation via diffusion. Default values follow the original paper settings.

How It Works

Emu (Sun et al., 2023) unifies visual understanding and generation in a single model. It uses an EVA-CLIP vision encoder to extract visual features, a Causal Transformer (LLaMA-based) as the multimodal LLM, and a visual regression head that maps LLM hidden states back to the visual embedding space for image generation via a diffusion decoder.

References:

  • Paper: “Generative Pretraining in Multimodality” (Sun et al., 2023)

Example

var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
var trainModel = new Emu<double>(architecture, new EmuOptions());

Constructors

Constructor Summary
Emu(NeuralNetworkArchitecture<T> architecture, EmuOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
Emu(NeuralNetworkArchitecture<T> architecture, string modelPath, EmuOptions options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates using Emu’s unified understanding + generation architecture.