ControlNetModel<T>

Models & Types · AiDotNet.Diffusion.Control

ControlNet model for adding spatial conditioning to diffusion models.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → ControlNetModel<T>

For Beginners

ControlNet is like giving the AI artist a reference sketch or blueprint to follow while creating an image.

Supported control types:

  • Canny edges: Outline/edge detection of shapes
  • Depth maps: 3D depth information
  • Pose keypoints: Human body positions (OpenPose)
  • Segmentation: Region/object boundaries
  • Normal maps: Surface orientation
  • Scribbles: Simple user drawings
  • Line art: Clean line drawings

How it works:

  1. You provide a control image (e.g., edge map of a house)
  2. ControlNet encodes this control signal
  3. The encoded control guides the diffusion process
  4. Result: Generated image follows the control structure

Example use cases:

  • “Draw a Victorian house” + edge map = house in exact shape
  • “Dancing woman” + pose skeleton = person in exact pose
  • “Forest scene” + depth map = correct 3D perspective

How It Works

ControlNet enables fine-grained spatial control over image generation by adding additional conditioning signals such as edge maps, depth maps, pose keypoints, segmentation masks, and more. It works by creating a trainable copy of the encoder blocks that process the control signal.

Technical details:

  • ControlNet is a “zero convolution” architecture
  • Copies encoder weights from base model
  • Adds control signal via residual connections
  • Can be combined: multi-ControlNet stacking
  • Supports conditioning strength adjustment (0-1)

Reference: Zhang et al., “Adding Conditional Control to Text-to-Image Diffusion Models”, 2023

Example

// Create a ControlNet model
var controlNet = new ControlNetModel<float>(
    controlType: ControlType.Canny);
            
// Generate with edge control
var edgeMap = new Tensor<float>([1, 3, 512, 512]);    // your Canny edge image
var image = controlNet.GenerateWithControl(
    prompt: "A beautiful Victorian house",
    controlImage: edgeMap,
    conditioningStrength: 1.0);
            
// Multi-control generation
var depthMap = new Tensor<float>([1, 3, 512, 512]);   // your depth map
var imageMulti = controlNet.GenerateWithMultiControl(
    prompt: "Forest landscape",
    controlImages: new[] { edgeMap, depthMap },
    controlTypes: new[] { ControlType.Canny, ControlType.Depth },
    conditioningStrengths: new[] { 0.8, 0.6 });

Constructors

Constructor Summary
ControlNetModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> baseUNet, StandardVAE<T> vae, IConditioningModule<T> conditioner, ControlType controlType, int? seed) Initializes a new instance of ControlNetModel with full customization support.

Properties

Property Summary
ControlType ControlType { get; } Gets the type of control signal this model uses.
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> VAE { get; }
double ConditioningStrength { get; set; } Gets or sets the default conditioning strength (0-1).
int LatentChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> GenerateWithControl(string prompt, Tensor<T> controlImage, string negativePrompt, int width, int height, int numInferenceSteps, double? guidanceScale, double? conditioningStrength, int? seed) Generates an image with control signal.
Tensor<T> GenerateWithMultiControl(string prompt, Tensor<T>[] controlImages, ControlType[] controlTypes, double[] conditioningStrengths, string negativePrompt, int width, int height, int numInferenceSteps, double? guidanceScale, int? seed) Generates an image with multiple control signals.