DiffEditModel<T>

Models & Types · AiDotNet.Diffusion.ImageEditing

DiffEdit model for automatic mask generation and text-guided image editing.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → DiffEditModel<T>

For Beginners

DiffEdit lets you edit images by just describing what to change.

How it works:

  1. You provide the original image, a source prompt, and a target prompt
  2. The model predicts noise for both prompts and computes the difference
  3. The difference map becomes an automatic editing mask
  4. The image is re-generated in masked regions guided by the target prompt

Key characteristics:

  • Based on Stable Diffusion 1.5 (512x512, CLIP ViT-L/14)
  • No manual mask required – masks are generated automatically
  • Preserves unedited regions faithfully
  • Uses DDIM scheduler for deterministic inversion and editing

How It Works

DiffEdit automatically generates editing masks by comparing noise predictions conditioned on source and target text prompts. The difference in predictions highlights which regions need to change, producing a spatial mask without any manual annotation. The mask is then used to selectively apply the edit while preserving unrelated regions.

Technical specifications:

  • Architecture: SD 1.5 U-Net with differential noise prediction for mask generation
  • Text encoder: CLIP ViT-L/14 (768-dim, 77 max tokens)
  • Cross-attention dimension: 768
  • VAE: 4 latent channels, scale factor 0.18215
  • Noise schedule: Scaled linear beta [0.00085, 0.012], 1000 timesteps
  • Scheduler: DDIM (deterministic inversion required)

Reference: Couairon et al., “DiffEdit: Diffusion-based Semantic Image Editing with Mask Guidance”, ICLR 2023

Example

var options = new DiffusionModelOptions<float> { LatentChannels = 4, DefaultInferenceSteps = 30 };
var input = Tensor<float>.CreateRandom(new[] { 1, 4, 64, 64 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new DiffEditModel<float>(options: options))
    .Build(trainX, trainY);
var edited = result.Predict(input);

Constructors

Constructor Summary
DiffEditModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> unet, StandardVAE<T> vae, IConditioningModule<T> conditioner, int? seed) Initializes a new instance of DiffEditModel with full customization support.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> VAE { get; }
int CrossAttentionDim { get; } Gets the cross-attention dimension (768 for CLIP ViT-L/14).
int LatentChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> GenerateFromText(string prompt, string negativePrompt, int width, int height, int numInferenceSteps, double? guidanceScale, int? seed)
Tensor<T> ImageToImage(Tensor<T> inputImage, string prompt, string negativePrompt, double strength, int numInferenceSteps, double? guidanceScale, int? seed)

Fields

Field Summary
int DefaultHeight Default image height for DiffEdit (SD 1.5 native resolution).
int DefaultWidth Default image width for DiffEdit (SD 1.5 native resolution).