UltraEditModel<T>

Models & Types · AiDotNet.Diffusion.ImageEditing

UltraEdit model for fine-grained instruction-based image editing with region awareness.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → UltraEditModel<T>

For Beginners

UltraEdit is a more precise version of text-based image editing. When you say “change the sky color,” it only modifies the sky and leaves everything else untouched, unlike simpler models that might accidentally change other parts too.

How It Works

UltraEdit improves upon InstructPix2Pix with region-aware editing that localizes changes to relevant areas. Uses automatically generated editing masks from LLM-based instruction parsing to prevent unwanted modifications to uninstructed regions. Built on SD1.5 backbone.

Technical specifications:

  • Base model: Stable Diffusion 1.5 (InstructPix2Pix variant)
  • Text encoder: CLIP ViT-L/14 (768-dim)
  • Input channels: 8 (4 latent + 4 source image latent)
  • Region awareness: LLM-generated editing masks from instructions
  • Training data: Large-scale instruction-image pairs with automatic masks

Reference: Zhao et al., “UltraEdit: Instruction-based Fine-Grained Image Editing at Scale”, 2024

Example

var options = new DiffusionModelOptions<float> { LatentChannels = 4, DefaultInferenceSteps = 30 };
var input = Tensor<float>.CreateRandom(new[] { 1, 4, 64, 64 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new UltraEditModel<float>(options: options))
    .Build(trainX, trainY);
var edited = result.Predict(input);

Constructors

Constructor Summary
UltraEditModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> predictor, StandardVAE<T> vae, IConditioningModule<T> conditioner, int? seed)

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> VAE { get; }
int LatentChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()