ReplaceAnythingModel<T>

Models & Types · AiDotNet.Diffusion.ImageEditing

ReplaceAnything model for text-guided object replacement within SAM-segmented regions.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → ReplaceAnythingModel<T>

For Beginners

Click on an object in your image (a car, a tree, a person), describe what you want instead (“a red sports car”, “a cherry blossom tree”), and ReplaceAnything seamlessly swaps it out while keeping the rest of the image intact.

How It Works

Combines SAM-based segmentation with SD1.5 inpainting diffusion to automatically replace any segmented object with text-described content. The user selects an object (via point, box, or mask) and provides a text description of the replacement.

Technical specifications:

  • Pipeline: SAM segmentation -> mask generation -> SD1.5 inpainting
  • Segmentation: Segment Anything Model (SAM) for interactive selection
  • Inpainting: SD1.5 with 9-channel input (4 latent + 4 masked + 1 mask)
  • Text encoder: CLIP ViT-L/14 (768-dim)
  • Selection modes: point, bounding box, or manual mask

Example

var options = new DiffusionModelOptions<float> { LatentChannels = 4, DefaultInferenceSteps = 30 };
var input = Tensor<float>.CreateRandom(new[] { 1, 4, 64, 64 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new ReplaceAnythingModel<float>(options: options))
    .Build(trainX, trainY);
var prediction = result.Predict(input);

Constructors

Constructor Summary
ReplaceAnythingModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> predictor, StandardVAE<T> vae, IConditioningModule<T> conditioner, int? seed)

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> VAE { get; }
int LatentChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()