SmartEdit<T>
Models & Types · AiDotNet.VisionLanguage.Editing
SmartEdit - complex instruction-based image editing with a multimodal LLM.
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → SmartEdit<T>
Implements: IImageEditingVLM<T>
For Beginners
SmartEdit edits a photo from an instruction that may need reasoning
- “remove the second object from the left” rather than “make it red”. A language model works out
what you meant while looking at the picture, then a diffusion model repeatedly removes noise, steered by that understanding, until the edit appears.
How It Works
Architecture, per Huang et al. SmartEdit is a LATENT DIFFUSION model. The paper’s starting point is InstructPix2Pix, whose weakness it names directly: such methods “often fail to produce satisfactory results in complex scenarios due to their dependence on the simple CLIP text encoder”. SmartEdit replaces that encoder with an MLLM (LLaVA) and adds a Bidirectional Interaction Module, which “enables comprehensive bidirectional information interactions between the input image and the MLLM” - so image and instruction inform each other before the diffusion model sees either.
It therefore keeps InstructPix2Pix’s StableDiffusion geometry: a 4-channel latent, and an 8-channel denoiser input where the encoded source image is concatenated to the noisy latent.
Why this derives from LatentDiffusionModelBase. It previously
derived from VisionLanguageModelBase and folded a flat list of layers in order, which
cannot express a U-Net’s skip connections or timestep conditioning, and made a sampler answer to
invariants that demand deterministic inference. The latent-diffusion base supplies the
scheduler, the sampling loop, the channel padding and the diffusion training objective.
Example
using AiDotNet.VisionLanguage.Editing;
// SmartEdit: complex instruction-based image editing. Every constructor argument has a default,
// so the paper configuration needs none of them.
var model = new SmartEdit<double>();
// Or start from its options and change what you need.
var configured = new SmartEdit<double>(options: new SmartEditOptions());
Constructors
| Constructor | Summary |
|---|---|
SmartEdit(NeuralNetworkArchitecture<T> architecture, SmartEditOptions options, DiffusionModelOptions<T> diffusionOptions, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> unet, StandardVAE<T> vae, int? seed) |
Creates a SmartEdit model. |
Properties
| Property | Summary |
|---|---|
IConditioningModule<T> Conditioner { get; } |
No CLIP text conditioner. |
INoisePredictor<T> NoisePredictor { get; } |
|
IVAEModel<T> VAE { get; } |
|
int EmbeddingDimension { get; } |
Width of the MLLM decoder whose tokens the interaction module consumes. |
int LatentChannels { get; } |
|
int OutputImageSize { get; } |
Edge length of the produced image. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
ModelOptions GetOptions() |
|
Tensor<T> EditImage(Tensor<T> image, string instruction) |
Edits image according to instruction. |
Tensor<T> EncodeImage(Tensor<T> image) |