BlendedDiffusionModel<T>

Models & Types · AiDotNet.Diffusion.ImageEditing

Blended Diffusion model for text-guided local image editing within user-specified masks.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → BlendedDiffusionModel<T>

For Beginners

Blended Diffusion lets you edit specific areas of an image with text.

How it works:

  1. You provide an image, a mask marking the area to edit, and a text prompt
  2. At each denoising step, new content is generated in the masked region
  3. The generated content is blended with the original image in unmasked regions
  4. The result is a seamless edit confined to the masked area

Key characteristics:

  • Based on Stable Diffusion 1.5 (512x512, CLIP ViT-L/14)
  • User-specified masks control where edits are applied
  • Per-step blending ensures seamless transitions
  • Uses DDPM scheduler for stochastic blending quality

How It Works

Blended Diffusion enables text-guided local editing by combining CLIP-guided diffusion with spatial blending. At each denoising step, the model generates content guided by the text prompt in masked regions and blends it with the original image content in unmasked regions, producing seamless localized edits.

Technical specifications:

  • Architecture: SD 1.5 U-Net with per-step spatial blending
  • Text encoder: CLIP ViT-L/14 (768-dim, 77 max tokens)
  • Cross-attention dimension: 768
  • VAE: 4 latent channels, scale factor 0.18215
  • Noise schedule: Scaled linear beta [0.00085, 0.012], 1000 timesteps
  • Scheduler: DDPM (stochastic for natural blending transitions)

Reference: Avrahami et al., “Blended Diffusion for Text-driven Editing of Natural Images”, CVPR 2022

Example

var options = new DiffusionModelOptions<float> { LatentChannels = 4, DefaultInferenceSteps = 30 };
var input = Tensor<float>.CreateRandom(new[] { 1, 4, 64, 64 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new BlendedDiffusionModel<float>(options: options))
    .Build(trainX, trainY);
var edited = result.Predict(input);

Constructors

Constructor Summary
BlendedDiffusionModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> unet, StandardVAE<T> vae, IConditioningModule<T> conditioner, int? seed) Initializes a new instance of BlendedDiffusionModel with full customization support.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> VAE { get; }
int CrossAttentionDim { get; } Gets the cross-attention dimension (768 for CLIP ViT-L/14).
int LatentChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> GenerateFromText(string prompt, string negativePrompt, int width, int height, int numInferenceSteps, double? guidanceScale, int? seed)
Tensor<T> ImageToImage(Tensor<T> inputImage, string prompt, string negativePrompt, double strength, int numInferenceSteps, double? guidanceScale, int? seed)

Fields

Field Summary
int DefaultHeight Default image height for Blended Diffusion (SD 1.5 native resolution).
int DefaultWidth Default image width for Blended Diffusion (SD 1.5 native resolution).