MagicBrushModel<T>

Models & Types · AiDotNet.Diffusion.ImageEditing

MagicBrush model for instruction-based image editing with visual brush stroke guidance.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → MagicBrushModel<T>

For Beginners

MagicBrush lets you edit images by describing what you want changed.

How it works:

  1. You provide a source image and a text instruction (e.g., “make the sky sunset-colored”)
  2. Optionally provide brush strokes to guide where edits should occur
  3. The model performs the described edit while preserving the rest of the image

Key characteristics:

  • Based on Stable Diffusion 1.5 (512x512, CLIP ViT-L/14)
  • Trained on large-scale human-annotated editing pairs
  • Supports both instruction-only and brush-guided editing
  • Uses Euler scheduler for fast, high-quality sampling

How It Works

MagicBrush enables instruction-guided image editing by combining natural language instructions with visual brush stroke annotations. It was trained on a large-scale dataset of manually annotated editing triplets (source image, instruction, target image) collected using DALL-E 2 and human annotators, allowing it to perform precise local and global edits.

Technical specifications:

  • Architecture: SD 1.5 U-Net fine-tuned for instruction-based editing
  • Text encoder: CLIP ViT-L/14 (768-dim, 77 max tokens)
  • Cross-attention dimension: 768
  • VAE: 4 latent channels, scale factor 0.18215
  • Noise schedule: Scaled linear beta [0.00085, 0.012], 1000 timesteps
  • Scheduler: Euler discrete

Reference: Zhang et al., “MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing”, NeurIPS 2024

Example

var options = new DiffusionModelOptions<float> { LatentChannels = 4, DefaultInferenceSteps = 30 };
var input = Tensor<float>.CreateRandom(new[] { 1, 4, 64, 64 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new MagicBrushModel<float>(options: options))
    .Build(trainX, trainY);
var edited = result.Predict(input);

Constructors

Constructor Summary
MagicBrushModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> unet, StandardVAE<T> vae, IConditioningModule<T> conditioner, int? seed) Initializes a new instance of MagicBrushModel with full customization support.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> VAE { get; }
int CrossAttentionDim { get; } Gets the cross-attention dimension (768 for CLIP ViT-L/14).
int LatentChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> GenerateFromText(string prompt, string negativePrompt, int width, int height, int numInferenceSteps, double? guidanceScale, int? seed)
Tensor<T> ImageToImage(Tensor<T> inputImage, string prompt, string negativePrompt, double strength, int numInferenceSteps, double? guidanceScale, int? seed)

Fields

Field Summary
int DefaultHeight Default image height for MagicBrush (SD 1.5 native resolution).
int DefaultWidth Default image width for MagicBrush (SD 1.5 native resolution).