UltraEditModel<T>
Models & Types · AiDotNet.Diffusion.ImageEditing
UltraEdit model for fine-grained instruction-based image editing with region awareness.
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → UltraEditModel<T>
For Beginners
UltraEdit is a more precise version of text-based image editing. When you say “change the sky color,” it only modifies the sky and leaves everything else untouched, unlike simpler models that might accidentally change other parts too.
How It Works
UltraEdit improves upon InstructPix2Pix with region-aware editing that localizes changes to relevant areas. Uses automatically generated editing masks from LLM-based instruction parsing to prevent unwanted modifications to uninstructed regions. Built on SD1.5 backbone.
Technical specifications:
- Base model: Stable Diffusion 1.5 (InstructPix2Pix variant)
- Text encoder: CLIP ViT-L/14 (768-dim)
- Input channels: 8 (4 latent + 4 source image latent)
- Region awareness: LLM-generated editing masks from instructions
- Training data: Large-scale instruction-image pairs with automatic masks
Reference: Zhao et al., “UltraEdit: Instruction-based Fine-Grained Image Editing at Scale”, 2024
Example
var options = new DiffusionModelOptions<float> { LatentChannels = 4, DefaultInferenceSteps = 30 };
var input = Tensor<float>.CreateRandom(new[] { 1, 4, 64, 64 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new UltraEditModel<float>(options: options))
.Build(trainX, trainY);
var edited = result.Predict(input);
Constructors
| Constructor | Summary |
|---|---|
UltraEditModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> predictor, StandardVAE<T> vae, IConditioningModule<T> conditioner, int? seed) |
Properties
| Property | Summary |
|---|---|
IConditioningModule<T> Conditioner { get; } |
|
INoisePredictor<T> NoisePredictor { get; } |
|
IVAEModel<T> VAE { get; } |
|
int LatentChannels { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |