MGIE<T>
Models & Types · AiDotNet.VisionLanguage.Editing
MGIE - instruction-based image editing guided by a multimodal large language model.
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → MGIE<T>
Implements: IImageEditingVLM<T>
For Beginners
MGIE edits a photo from a written instruction. A language model first works out what you meant and records it as a compact hint; a diffusion model then repeatedly removes noise from the picture, steered by that hint, until the edit appears.
How It Works
Architecture, per Fu et al. Sec. 3. MGIE is a LATENT DIFFUSION model:
Why this derives from LatentDiffusionModelBase. It previously
derived from VisionLanguageModelBase and folded a flat list of layers in order. That
shape cannot express the paper’s model: a U-Net needs skip connections and timestep
conditioning, and denoising is a loop rather than one forward pass. It also meant the model was
checked by neural-network invariants that demand deterministic inference, which no sampler can
satisfy. Deriving from the latent-diffusion base puts MGIE alongside its siblings in
AiDotNet.Diffusion.ImageEditing - among them InstructPix2PixModel, which this
paper builds directly on - and it inherits the scheduler, the sampling loop and the diffusion
training objective instead of approximating them.
Example
using AiDotNet.VisionLanguage.Editing;
// MGIE: image editing guided by a multimodal LLM. Every constructor argument has a default,
// so the paper configuration needs none of them.
var model = new MGIE<double>();
// Or start from its options and change what you need.
var configured = new MGIE<double>(options: new MGIEOptions());
Constructors
| Constructor | Summary |
|---|---|
MGIE(LLaVANeuralNetwork<T> instructionEncoder, NeuralNetworkArchitecture<T> architecture, MGIEOptions options, DiffusionModelOptions<T> diffusionOptions, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> unet, StandardVAE<T> vae, int? seed) |
Creates an MGIE model with a native instruction encoder and its matching tokenizer. |
MGIE(NeuralNetworkArchitecture<T> architecture, MGIEOptions options, DiffusionModelOptions<T> diffusionOptions, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> unet, StandardVAE<T> vae, int? seed) |
Creates an MGIE model. |
Properties
| Property | Summary |
|---|---|
IConditioningModule<T> Conditioner { get; } |
No separate text conditioner. |
INoisePredictor<T> NoisePredictor { get; } |
|
IVAEModel<T> VAE { get; } |
|
double GuidanceScale { get; } |
|
int EmbeddingDimension { get; } |
Width of the MLLM decoder whose [IMG] tokens the edit head consumes. |
int LatentChannels { get; } |
|
int OutputImageSize { get; } |
Edge length of the produced image. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
ModelOptions GetOptions() |
|
T TrainEdit(Tensor<T> sourceImage, IReadOnlyList<int> instructionTokenIds, Tensor<T> editedImage, IReadOnlyList<int> expressiveInstructionTokenIds) |
Runs one MGIE training step on an image, an instruction and the edited target image. |
Tensor<T> EditImage(Tensor<T> image, string instruction) |
|
Tensor<T> EditImage(Tensor<T> image, string instruction, int? seed, Vector<T> standardNoise) |
Edits an RGB image in [-1,1] using seeded fresh target noise and the source VAE’s unscaled mode. |
Tensor<T> EncodeEditGuidance(Tensor<T> image, IReadOnlyList<int> instructionTokenIds) |
Encodes an explicit image/instruction token sequence into trainable diffusion context. |
Tensor<T> EncodeImage(Tensor<T> image) |
Returns visual tokens in the language model’s feature space, not VAE latents. |