SDEditModel<T>

Models & Types · AiDotNet.Diffusion.ImageEditing

SDEdit model for image-guided synthesis via partial noise injection and guided denoising.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → SDEditModel<T>

For Beginners

SDEdit turns rough sketches and images into polished results.

How it works:

  1. You provide an input image (sketch, rough edit, or photograph)
  2. The model adds noise to partially corrupt the image
  3. The diffusion model denoises it, guided by text prompts
  4. The result balances input structure with photorealistic output

Key characteristics:

  • Based on Stable Diffusion 1.5 (512x512, CLIP ViT-L/14)
  • No fine-tuning needed – works with any pre-trained diffusion model
  • Noise strength controls realism vs. faithfulness tradeoff
  • Uses DDPM scheduler for stochastic denoising

How It Works

SDEdit (Stochastic Differential Editing) transforms sketches, strokes, and existing images into realistic outputs by adding controlled amounts of noise to the input and then denoising with a diffusion model. The noise strength controls the balance between faithfulness to the input guide and realism of the output.

Technical specifications:

  • Architecture: SD 1.5 U-Net with partial noise injection pipeline
  • Text encoder: CLIP ViT-L/14 (768-dim, 77 max tokens)
  • Cross-attention dimension: 768
  • VAE: 4 latent channels, scale factor 0.18215
  • Noise schedule: Scaled linear beta [0.00085, 0.012], 1000 timesteps
  • Scheduler: DDPM (stochastic denoising for natural output variation)

Reference: Meng et al., “SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations”, ICLR 2022

Example

var options = new DiffusionModelOptions<float> { LatentChannels = 4, DefaultInferenceSteps = 30 };
var input = Tensor<float>.CreateRandom(new[] { 1, 4, 64, 64 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new SDEditModel<float>(options: options))
    .Build(trainX, trainY);
var edited = result.Predict(input);

Constructors

Constructor Summary
SDEditModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> unet, StandardVAE<T> vae, IConditioningModule<T> conditioner, int? seed) Initializes a new instance of SDEditModel with full customization support.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> VAE { get; }
int CrossAttentionDim { get; } Gets the cross-attention dimension (768 for CLIP ViT-L/14).
int LatentChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> GenerateFromText(string prompt, string negativePrompt, int width, int height, int numInferenceSteps, double? guidanceScale, int? seed)
Tensor<T> ImageToImage(Tensor<T> inputImage, string prompt, string negativePrompt, double strength, int numInferenceSteps, double? guidanceScale, int? seed)

Fields

Field Summary
int DefaultHeight Default image height for SDEdit (SD 1.5 native resolution).
int DefaultWidth Default image width for SDEdit (SD 1.5 native resolution).