ImagicModel<T>

Models & Types · AiDotNet.Diffusion.ImageEditing

Imagic model for text-aligned real image editing via embedding optimization and model fine-tuning.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → ImagicModel<T>

For Beginners

Imagic edits real photos to match your text description.

How it works:

  1. A text embedding is optimized to faithfully reconstruct the original image
  2. The diffusion model is fine-tuned to bind the embedding to the image content
  3. Interpolation between optimized and target embeddings applies the desired edit

Key characteristics:

  • Based on Stable Diffusion 1.5 (512x512, CLIP ViT-L/14)
  • Can perform complex edits (pose changes, object addition, style transfer)
  • Requires per-image optimization and fine-tuning
  • Uses DDIM scheduler for deterministic generation

How It Works

Imagic enables sophisticated editing of real images to match a target text description through a three-stage process: (1) optimizing a text embedding to reconstruct the input image, (2) fine-tuning the diffusion model with the optimized embedding, and (3) interpolating between the optimized and target embeddings to produce the desired edit.

Technical specifications:

  • Architecture: SD 1.5 U-Net with text embedding optimization + model fine-tuning
  • Text encoder: CLIP ViT-L/14 (768-dim, 77 max tokens)
  • Cross-attention dimension: 768
  • VAE: 4 latent channels, scale factor 0.18215
  • Noise schedule: Scaled linear beta [0.00085, 0.012], 1000 timesteps
  • Scheduler: DDIM (deterministic for consistent interpolation results)

Reference: Kawar et al., “Imagic: Text-Based Real Image Editing with Diffusion Models”, CVPR 2023

Example

var options = new DiffusionModelOptions<float> { LatentChannels = 4, DefaultInferenceSteps = 30 };
var input = Tensor<float>.CreateRandom(new[] { 1, 4, 64, 64 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new ImagicModel<float>(options: options))
    .Build(trainX, trainY);
var edited = result.Predict(input);

Constructors

Constructor Summary
ImagicModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> unet, StandardVAE<T> vae, IConditioningModule<T> conditioner, int? seed) Initializes a new instance of ImagicModel with full customization support.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> VAE { get; }
int CrossAttentionDim { get; } Gets the cross-attention dimension (768 for CLIP ViT-L/14).
int LatentChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> GenerateFromText(string prompt, string negativePrompt, int width, int height, int numInferenceSteps, double? guidanceScale, int? seed)
Tensor<T> ImageToImage(Tensor<T> inputImage, string prompt, string negativePrompt, double strength, int numInferenceSteps, double? guidanceScale, int? seed)

Fields

Field Summary
int DefaultHeight Default image height for Imagic (SD 1.5 native resolution).
int DefaultWidth Default image width for Imagic (SD 1.5 native resolution).