Emu2<T>

Models & Types · AiDotNet.VisionLanguage.Generative

Emu2: scaled 37B unified understanding and generation model.

Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → Emu2<T>

Implements: IGenerativeVisionLanguageModel<T>

For Beginners

Emu2 scales the original Emu architecture to 37 billion parameters, using a larger EVA-CLIP-E vision encoder and a more powerful LLaMA-based decoder. It excels at in-context learning for multimodal tasks — given a few examples of image-text pairs, it can generalize to new tasks without fine-tuning. Default values follow the original paper settings.

How It Works

Emu2 (Sun et al., 2024) scales the Emu architecture to 37B parameters by using a larger EVA-CLIP-E encoder and a more powerful LLaMA-based decoder. It demonstrates strong in-context learning for multimodal tasks and can generate both text and images from interleaved inputs.

References:

  • Paper: “Generative Multimodal Models are In-Context Learners” (Sun et al., 2024)

Example

var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.TwoDimensional,
    taskType: NeuralNetworkTaskType.ImageClassification,
    inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
var trainModel = new Emu2<double>(architecture, new Emu2Options());

Constructors

Constructor Summary
Emu2(NeuralNetworkArchitecture<T> architecture, Emu2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
Emu2(NeuralNetworkArchitecture<T> architecture, string modelPath, Emu2Options options)

Properties

Property Summary
int DecoderEmbeddingDim { get; }
int EmbeddingDimension { get; }
int MaxGenerationLength { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> EncodeImage(Tensor<T> image)
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) Generates using Emu2’s scaled 37B unified architecture.