BrushEditModel<T>

Models & Types · AiDotNet.Diffusion.ImageEditing

BrushEdit model combining multimodal LLM understanding with BrushNet inpainting.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → BrushEditModel<T>

For Beginners

BrushEdit combines AI understanding with inpainting. Instead of drawing masks yourself, you just describe what to change (“remove the car on the left”), and BrushEdit figures out where to mask and what to generate.

How It Works

BrushEdit uses a multimodal LLM to understand editing instructions and automatically generate appropriate masks, then applies BrushNet-style dual-branch inpainting on SD1.5. This enables natural language editing without manual mask creation.

Technical specifications:

  • Pipeline: Multimodal LLM (mask generation) + BrushNet (dual-branch inpainting)
  • Inpainting backbone: SD1.5 with BrushNet dual-branch architecture
  • Text encoder: CLIP ViT-L/14 (768-dim)
  • Input channels: 9 (4 latent + 4 masked image latent + 1 mask)
  • Mask generation: Automatic from natural language via LLM

Reference: Li et al., “BrushEdit: All-In-One Image Inpainting and Editing”, 2024

Example

var options = new DiffusionModelOptions<float> { LatentChannels = 4, DefaultInferenceSteps = 30 };
var input = Tensor<float>.CreateRandom(new[] { 1, 4, 64, 64 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new BrushEditModel<float>(options: options))
    .Build(trainX, trainY);
var edited = result.Predict(input);

Constructors

Constructor Summary
BrushEditModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> predictor, StandardVAE<T> vae, IConditioningModule<T> conditioner, int? seed)

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> VAE { get; }
int LatentChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()