MGIE<T>

Models & Types · AiDotNet.VisionLanguage.Editing

MGIE - instruction-based image editing guided by a multimodal large language model.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → MGIE<T>

Implements: IImageEditingVLM<T>

For Beginners

MGIE edits a photo from a written instruction. A language model first works out what you meant and records it as a compact hint; a diffusion model then repeatedly removes noise from the picture, steered by that hint, until the edit appears.

How It Works

Architecture, per Fu et al. Sec. 3. MGIE is a LATENT DIFFUSION model:

Why this derives from LatentDiffusionModelBase. It previously derived from VisionLanguageModelBase and folded a flat list of layers in order. That shape cannot express the paper’s model: a U-Net needs skip connections and timestep conditioning, and denoising is a loop rather than one forward pass. It also meant the model was checked by neural-network invariants that demand deterministic inference, which no sampler can satisfy. Deriving from the latent-diffusion base puts MGIE alongside its siblings in AiDotNet.Diffusion.ImageEditing - among them InstructPix2PixModel, which this paper builds directly on - and it inherits the scheduler, the sampling loop and the diffusion training objective instead of approximating them.

Example

using AiDotNet.VisionLanguage.Editing;
// MGIE: image editing guided by a multimodal LLM. Every constructor argument has a default,
// so the paper configuration needs none of them.
var model = new MGIE<double>();
            
// Or start from its options and change what you need.
var configured = new MGIE<double>(options: new MGIEOptions());

Constructors

Constructor Summary
MGIE(LLaVANeuralNetwork<T> instructionEncoder, NeuralNetworkArchitecture<T> architecture, MGIEOptions options, DiffusionModelOptions<T> diffusionOptions, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> unet, StandardVAE<T> vae, int? seed) Creates an MGIE model with a native instruction encoder and its matching tokenizer.
MGIE(NeuralNetworkArchitecture<T> architecture, MGIEOptions options, DiffusionModelOptions<T> diffusionOptions, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> unet, StandardVAE<T> vae, int? seed) Creates an MGIE model.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; } No separate text conditioner.
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> VAE { get; }
double GuidanceScale { get; }
int EmbeddingDimension { get; } Width of the MLLM decoder whose [IMG] tokens the edit head consumes.
int LatentChannels { get; }
int OutputImageSize { get; } Edge length of the produced image.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
ModelOptions GetOptions()
T TrainEdit(Tensor<T> sourceImage, IReadOnlyList<int> instructionTokenIds, Tensor<T> editedImage, IReadOnlyList<int> expressiveInstructionTokenIds) Runs one MGIE training step on an image, an instruction and the edited target image.
Tensor<T> EditImage(Tensor<T> image, string instruction)
Tensor<T> EditImage(Tensor<T> image, string instruction, int? seed, Vector<T> standardNoise) Edits an RGB image in [-1,1] using seeded fresh target noise and the source VAE’s unscaled mode.
Tensor<T> EncodeEditGuidance(Tensor<T> image, IReadOnlyList<int> instructionTokenIds) Encodes an explicit image/instruction token sequence into trainable diffusion context.
Tensor<T> EncodeImage(Tensor<T> image) Returns visual tokens in the language model’s feature space, not VAE latents.