DiffEditModel<T>
Models & Types · AiDotNet.Diffusion.ImageEditing
DiffEdit model for automatic mask generation and text-guided image editing.
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → DiffEditModel<T>
For Beginners
DiffEdit lets you edit images by just describing what to change.
How it works:
- You provide the original image, a source prompt, and a target prompt
- The model predicts noise for both prompts and computes the difference
- The difference map becomes an automatic editing mask
- The image is re-generated in masked regions guided by the target prompt
Key characteristics:
- Based on Stable Diffusion 1.5 (512x512, CLIP ViT-L/14)
- No manual mask required – masks are generated automatically
- Preserves unedited regions faithfully
- Uses DDIM scheduler for deterministic inversion and editing
How It Works
DiffEdit automatically generates editing masks by comparing noise predictions conditioned on source and target text prompts. The difference in predictions highlights which regions need to change, producing a spatial mask without any manual annotation. The mask is then used to selectively apply the edit while preserving unrelated regions.
Technical specifications:
- Architecture: SD 1.5 U-Net with differential noise prediction for mask generation
- Text encoder: CLIP ViT-L/14 (768-dim, 77 max tokens)
- Cross-attention dimension: 768
- VAE: 4 latent channels, scale factor 0.18215
- Noise schedule: Scaled linear beta [0.00085, 0.012], 1000 timesteps
- Scheduler: DDIM (deterministic inversion required)
Reference: Couairon et al., “DiffEdit: Diffusion-based Semantic Image Editing with Mask Guidance”, ICLR 2023
Example
var options = new DiffusionModelOptions<float> { LatentChannels = 4, DefaultInferenceSteps = 30 };
var input = Tensor<float>.CreateRandom(new[] { 1, 4, 64, 64 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new DiffEditModel<float>(options: options))
.Build(trainX, trainY);
var edited = result.Predict(input);
Constructors
| Constructor | Summary |
|---|---|
DiffEditModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> unet, StandardVAE<T> vae, IConditioningModule<T> conditioner, int? seed) |
Initializes a new instance of DiffEditModel with full customization support. |
Properties
| Property | Summary |
|---|---|
IConditioningModule<T> Conditioner { get; } |
|
INoisePredictor<T> NoisePredictor { get; } |
|
IVAEModel<T> VAE { get; } |
|
int CrossAttentionDim { get; } |
Gets the cross-attention dimension (768 for CLIP ViT-L/14). |
int LatentChannels { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> GenerateFromText(string prompt, string negativePrompt, int width, int height, int numInferenceSteps, double? guidanceScale, int? seed) |
|
Tensor<T> ImageToImage(Tensor<T> inputImage, string prompt, string negativePrompt, double strength, int numInferenceSteps, double? guidanceScale, int? seed) |
Fields
| Field | Summary |
|---|---|
int DefaultHeight |
Default image height for DiffEdit (SD 1.5 native resolution). |
int DefaultWidth |
Default image width for DiffEdit (SD 1.5 native resolution). |