EmuEdit<T>

Models & Types · AiDotNet.VisionLanguage.Editing

Emu Edit - precise image editing via recognition and generation tasks.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → EmuEdit<T>

Implements: IImageEditingVLM<T>

For Beginners

Emu Edit changes a photo according to a written instruction. It first works out what KIND of edit you are asking for - remove something, change a colour, add an object - and uses that judgement to steer a diffusion model as it repeatedly removes noise from the image.

How It Works

Architecture, per Sheynin et al. Emu Edit is a LATENT DIFFUSION model built on Emu, which “incorporated a 16-channel autoencoder with encoder E and decoder D” and generates at 512x512. That is the one place this model departs from its SmartEdit and MGIE siblings, which inherit StableDiffusion’s 4-channel latent: Emu’s autoencoder is deliberately wider.

Its distinguishing mechanism is the learned task embedding. The paper: “for each task, we learn a unique task embedding vector, and integrate it into the model through cross-attention interactions, and by adding it to the timestep embedding” - so a single model infers the right edit type from a free-form instruction rather than being told.

Why this derives from LatentDiffusionModelBase. It previously derived from VisionLanguageModelBase and folded a flat list of layers in order, which cannot express a U-Net’s skip connections or timestep conditioning - and timestep conditioning is exactly where this model’s task embedding lives, so the old shape could not represent its central idea. The latent-diffusion base supplies the scheduler, the sampling loop, the channel padding and the diffusion training objective.

Example

using AiDotNet.VisionLanguage.Editing;
// Emu Edit: instruction-based image editing on a latent-diffusion backbone. Every constructor argument has a default,
// so the paper configuration needs none of them.
var model = new EmuEdit<double>();
            
// Or start from its options and change what you need.
var configured = new EmuEdit<double>(options: new EmuEditOptions());

Constructors

Constructor Summary
EmuEdit(NeuralNetworkArchitecture<T> architecture, EmuEditOptions options, DiffusionModelOptions<T> diffusionOptions, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> unet, StandardVAE<T> vae, int? seed) Creates an Emu Edit model.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; } No separate text conditioner: the learned task embedding is what steers generation toward the correct edit type, and it reaches the denoiser through cross-attention and the timestep embedding rather than through a text-encoder module.
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> VAE { get; }
int EmbeddingDimension { get; } Width of the decoder whose tokens the edit head consumes.
int LatentChannels { get; }
int OutputImageSize { get; } Edge length of the produced image.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
ModelOptions GetOptions()
Tensor<T> EditImage(Tensor<T> image, string instruction) Edits image according to instruction.
Tensor<T> EncodeImage(Tensor<T> image)