BlendedDiffusionModel<T>
Models & Types · AiDotNet.Diffusion.ImageEditing
Blended Diffusion model for text-guided local image editing within user-specified masks.
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → BlendedDiffusionModel<T>
For Beginners
Blended Diffusion lets you edit specific areas of an image with text.
How it works:
- You provide an image, a mask marking the area to edit, and a text prompt
- At each denoising step, new content is generated in the masked region
- The generated content is blended with the original image in unmasked regions
- The result is a seamless edit confined to the masked area
Key characteristics:
- Based on Stable Diffusion 1.5 (512x512, CLIP ViT-L/14)
- User-specified masks control where edits are applied
- Per-step blending ensures seamless transitions
- Uses DDPM scheduler for stochastic blending quality
How It Works
Blended Diffusion enables text-guided local editing by combining CLIP-guided diffusion with spatial blending. At each denoising step, the model generates content guided by the text prompt in masked regions and blends it with the original image content in unmasked regions, producing seamless localized edits.
Technical specifications:
- Architecture: SD 1.5 U-Net with per-step spatial blending
- Text encoder: CLIP ViT-L/14 (768-dim, 77 max tokens)
- Cross-attention dimension: 768
- VAE: 4 latent channels, scale factor 0.18215
- Noise schedule: Scaled linear beta [0.00085, 0.012], 1000 timesteps
- Scheduler: DDPM (stochastic for natural blending transitions)
Reference: Avrahami et al., “Blended Diffusion for Text-driven Editing of Natural Images”, CVPR 2022
Example
var options = new DiffusionModelOptions<float> { LatentChannels = 4, DefaultInferenceSteps = 30 };
var input = Tensor<float>.CreateRandom(new[] { 1, 4, 64, 64 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new BlendedDiffusionModel<float>(options: options))
.Build(trainX, trainY);
var edited = result.Predict(input);
Constructors
| Constructor | Summary |
|---|---|
BlendedDiffusionModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> unet, StandardVAE<T> vae, IConditioningModule<T> conditioner, int? seed) |
Initializes a new instance of BlendedDiffusionModel with full customization support. |
Properties
| Property | Summary |
|---|---|
IConditioningModule<T> Conditioner { get; } |
|
INoisePredictor<T> NoisePredictor { get; } |
|
IVAEModel<T> VAE { get; } |
|
int CrossAttentionDim { get; } |
Gets the cross-attention dimension (768 for CLIP ViT-L/14). |
int LatentChannels { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> GenerateFromText(string prompt, string negativePrompt, int width, int height, int numInferenceSteps, double? guidanceScale, int? seed) |
|
Tensor<T> ImageToImage(Tensor<T> inputImage, string prompt, string negativePrompt, double strength, int numInferenceSteps, double? guidanceScale, int? seed) |
Fields
| Field | Summary |
|---|---|
int DefaultHeight |
Default image height for Blended Diffusion (SD 1.5 native resolution). |
int DefaultWidth |
Default image width for Blended Diffusion (SD 1.5 native resolution). |