PowerPaintModel<T>
Models & Types · AiDotNet.Diffusion.ImageEditing
PowerPaint v2 model for versatile task-aware image inpainting with learnable task prompts.
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → PowerPaintModel<T>
For Beginners
PowerPaint is a versatile inpainting model that can handle many different tasks: removing objects, inserting new objects, changing shapes, or extending images. You just tell it which task you want, and it adapts automatically through special learned task tokens.
How It Works
PowerPaint uses learnable task prompts to handle multiple inpainting tasks including text-guided object insertion, object removal, shape-guided generation, and outpainting with a single model. Task-specific prompt tokens are prepended to guide behavior. Built on Stable Diffusion 2.1 with OpenCLIP ViT-H text encoder.
Technical specifications:
- Base model: Stable Diffusion 2.1 (UNet)
- Text encoder: OpenCLIP ViT-H/14 (1024-dim context)
- Input channels: 9 (4 latent + 4 masked image + 1 mask)
- Task prompts: 4 learnable tokens per task type
- Tasks: text-guided insertion, removal, shape-guided, outpainting
- Resolution: 512x512 native
Reference: Zhuang et al., “A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpainting”, 2024
Example
var options = new DiffusionModelOptions<float> { LatentChannels = 4, DefaultInferenceSteps = 30 };
var input = Tensor<float>.CreateRandom(new[] { 1, 4, 64, 64 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new PowerPaintModel<float>(options: options))
.Build(trainX, trainY);
var inpainted = result.Predict(input);
Constructors
| Constructor | Summary |
|---|---|
PowerPaintModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> predictor, StandardVAE<T> vae, IConditioningModule<T> conditioner, int? seed) |
Properties
| Property | Summary |
|---|---|
IConditioningModule<T> Conditioner { get; } |
|
INoisePredictor<T> NoisePredictor { get; } |
|
IVAEModel<T> VAE { get; } |
|
int LatentChannels { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |