Emu2<T>
Models & Types · AiDotNet.VisionLanguage.Generative
Emu2: scaled 37B unified understanding and generation model.
Inheritance: NeuralNetworkBase<T> → VisionLanguageModelBase<T> → Emu2<T>
Implements: IGenerativeVisionLanguageModel<T>
For Beginners
Emu2 scales the original Emu architecture to 37 billion parameters, using a larger EVA-CLIP-E vision encoder and a more powerful LLaMA-based decoder. It excels at in-context learning for multimodal tasks — given a few examples of image-text pairs, it can generalize to new tasks without fine-tuning. Default values follow the original paper settings.
How It Works
Emu2 (Sun et al., 2024) scales the Emu architecture to 37B parameters by using a larger EVA-CLIP-E encoder and a more powerful LLaMA-based decoder. It demonstrates strong in-context learning for multimodal tasks and can generate both text and images from interleaved inputs.
References:
- Paper: “Generative Multimodal Models are In-Context Learners” (Sun et al., 2024)
Example
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.TwoDimensional,
taskType: NeuralNetworkTaskType.ImageClassification,
inputHeight: 224, inputWidth: 224, inputDepth: 3, outputSize: 512);
var trainModel = new Emu2<double>(architecture, new Emu2Options());
Constructors
| Constructor | Summary |
|---|---|
Emu2(NeuralNetworkArchitecture<T> architecture, Emu2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
Emu2(NeuralNetworkArchitecture<T> architecture, string modelPath, Emu2Options options) |
Properties
| Property | Summary |
|---|---|
int DecoderEmbeddingDim { get; } |
|
int EmbeddingDimension { get; } |
|
int MaxGenerationLength { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> EncodeImage(Tensor<T> image) |
|
Tensor<T> GenerateFromImage(Tensor<T> image, string prompt) |
Generates using Emu2’s scaled 37B unified architecture. |