SmartEdit<T>

Models & Types · AiDotNet.VisionLanguage.Editing

SmartEdit - complex instruction-based image editing with a multimodal LLM.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → SmartEdit<T>

Implements: IImageEditingVLM<T>

For Beginners

SmartEdit edits a photo from an instruction that may need reasoning

  • “remove the second object from the left” rather than “make it red”. A language model works out

what you meant while looking at the picture, then a diffusion model repeatedly removes noise, steered by that understanding, until the edit appears.

How It Works

Architecture, per Huang et al. SmartEdit is a LATENT DIFFUSION model. The paper’s starting point is InstructPix2Pix, whose weakness it names directly: such methods “often fail to produce satisfactory results in complex scenarios due to their dependence on the simple CLIP text encoder”. SmartEdit replaces that encoder with an MLLM (LLaVA) and adds a Bidirectional Interaction Module, which “enables comprehensive bidirectional information interactions between the input image and the MLLM” - so image and instruction inform each other before the diffusion model sees either.

It therefore keeps InstructPix2Pix’s StableDiffusion geometry: a 4-channel latent, and an 8-channel denoiser input where the encoded source image is concatenated to the noisy latent.

Why this derives from LatentDiffusionModelBase. It previously derived from VisionLanguageModelBase and folded a flat list of layers in order, which cannot express a U-Net’s skip connections or timestep conditioning, and made a sampler answer to invariants that demand deterministic inference. The latent-diffusion base supplies the scheduler, the sampling loop, the channel padding and the diffusion training objective.

Example

using AiDotNet.VisionLanguage.Editing;
// SmartEdit: complex instruction-based image editing. Every constructor argument has a default,
// so the paper configuration needs none of them.
var model = new SmartEdit<double>();
            
// Or start from its options and change what you need.
var configured = new SmartEdit<double>(options: new SmartEditOptions());

Constructors

Constructor Summary
SmartEdit(NeuralNetworkArchitecture<T> architecture, SmartEditOptions options, DiffusionModelOptions<T> diffusionOptions, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> unet, StandardVAE<T> vae, int? seed) Creates a SmartEdit model.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; } No CLIP text conditioner.
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> VAE { get; }
int EmbeddingDimension { get; } Width of the MLLM decoder whose tokens the interaction module consumes.
int LatentChannels { get; }
int OutputImageSize { get; } Edge length of the produced image.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
ModelOptions GetOptions()
Tensor<T> EditImage(Tensor<T> image, string instruction) Edits image according to instruction.
Tensor<T> EncodeImage(Tensor<T> image)