EmuEdit<T>
Models & Types · AiDotNet.VisionLanguage.Editing
Emu Edit - precise image editing via recognition and generation tasks.
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → EmuEdit<T>
Implements: IImageEditingVLM<T>
For Beginners
Emu Edit changes a photo according to a written instruction. It first works out what KIND of edit you are asking for - remove something, change a colour, add an object - and uses that judgement to steer a diffusion model as it repeatedly removes noise from the image.
How It Works
Architecture, per Sheynin et al. Emu Edit is a LATENT DIFFUSION model built on Emu, which “incorporated a 16-channel autoencoder with encoder E and decoder D” and generates at 512x512. That is the one place this model departs from its SmartEdit and MGIE siblings, which inherit StableDiffusion’s 4-channel latent: Emu’s autoencoder is deliberately wider.
Its distinguishing mechanism is the learned task embedding. The paper: “for each task, we learn a unique task embedding vector, and integrate it into the model through cross-attention interactions, and by adding it to the timestep embedding” - so a single model infers the right edit type from a free-form instruction rather than being told.
Why this derives from LatentDiffusionModelBase. It previously
derived from VisionLanguageModelBase and folded a flat list of layers in order, which
cannot express a U-Net’s skip connections or timestep conditioning - and timestep conditioning
is exactly where this model’s task embedding lives, so the old shape could not represent its
central idea. The latent-diffusion base supplies the scheduler, the sampling loop, the channel
padding and the diffusion training objective.
Example
using AiDotNet.VisionLanguage.Editing;
// Emu Edit: instruction-based image editing on a latent-diffusion backbone. Every constructor argument has a default,
// so the paper configuration needs none of them.
var model = new EmuEdit<double>();
// Or start from its options and change what you need.
var configured = new EmuEdit<double>(options: new EmuEditOptions());
Constructors
| Constructor | Summary |
|---|---|
EmuEdit(NeuralNetworkArchitecture<T> architecture, EmuEditOptions options, DiffusionModelOptions<T> diffusionOptions, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> unet, StandardVAE<T> vae, int? seed) |
Creates an Emu Edit model. |
Properties
| Property | Summary |
|---|---|
IConditioningModule<T> Conditioner { get; } |
No separate text conditioner: the learned task embedding is what steers generation toward the correct edit type, and it reaches the denoiser through cross-attention and the timestep embedding rather than through a text-encoder module. |
INoisePredictor<T> NoisePredictor { get; } |
|
IVAEModel<T> VAE { get; } |
|
int EmbeddingDimension { get; } |
Width of the decoder whose tokens the edit head consumes. |
int LatentChannels { get; } |
|
int OutputImageSize { get; } |
Edge length of the produced image. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
ModelOptions GetOptions() |
|
Tensor<T> EditImage(Tensor<T> image, string instruction) |
Edits image according to instruction. |
Tensor<T> EncodeImage(Tensor<T> image) |