ControlNetModel<T>
Models & Types · AiDotNet.Diffusion.Control
ControlNet model for adding spatial conditioning to diffusion models.
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → ControlNetModel<T>
For Beginners
ControlNet is like giving the AI artist a reference sketch or blueprint to follow while creating an image.
Supported control types:
- Canny edges: Outline/edge detection of shapes
- Depth maps: 3D depth information
- Pose keypoints: Human body positions (OpenPose)
- Segmentation: Region/object boundaries
- Normal maps: Surface orientation
- Scribbles: Simple user drawings
- Line art: Clean line drawings
How it works:
- You provide a control image (e.g., edge map of a house)
- ControlNet encodes this control signal
- The encoded control guides the diffusion process
- Result: Generated image follows the control structure
Example use cases:
- “Draw a Victorian house” + edge map = house in exact shape
- “Dancing woman” + pose skeleton = person in exact pose
- “Forest scene” + depth map = correct 3D perspective
How It Works
ControlNet enables fine-grained spatial control over image generation by adding additional conditioning signals such as edge maps, depth maps, pose keypoints, segmentation masks, and more. It works by creating a trainable copy of the encoder blocks that process the control signal.
Technical details:
- ControlNet is a “zero convolution” architecture
- Copies encoder weights from base model
- Adds control signal via residual connections
- Can be combined: multi-ControlNet stacking
- Supports conditioning strength adjustment (0-1)
Reference: Zhang et al., “Adding Conditional Control to Text-to-Image Diffusion Models”, 2023
Example
// Create a ControlNet model
var controlNet = new ControlNetModel<float>(
controlType: ControlType.Canny);
// Generate with edge control
var edgeMap = new Tensor<float>([1, 3, 512, 512]); // your Canny edge image
var image = controlNet.GenerateWithControl(
prompt: "A beautiful Victorian house",
controlImage: edgeMap,
conditioningStrength: 1.0);
// Multi-control generation
var depthMap = new Tensor<float>([1, 3, 512, 512]); // your depth map
var imageMulti = controlNet.GenerateWithMultiControl(
prompt: "Forest landscape",
controlImages: new[] { edgeMap, depthMap },
controlTypes: new[] { ControlType.Canny, ControlType.Depth },
conditioningStrengths: new[] { 0.8, 0.6 });
Constructors
| Constructor | Summary |
|---|---|
ControlNetModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> baseUNet, StandardVAE<T> vae, IConditioningModule<T> conditioner, ControlType controlType, int? seed) |
Initializes a new instance of ControlNetModel with full customization support. |
Properties
| Property | Summary |
|---|---|
ControlType ControlType { get; } |
Gets the type of control signal this model uses. |
IConditioningModule<T> Conditioner { get; } |
|
INoisePredictor<T> NoisePredictor { get; } |
|
IVAEModel<T> VAE { get; } |
|
double ConditioningStrength { get; set; } |
Gets or sets the default conditioning strength (0-1). |
int LatentChannels { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> GenerateWithControl(string prompt, Tensor<T> controlImage, string negativePrompt, int width, int height, int numInferenceSteps, double? guidanceScale, double? conditioningStrength, int? seed) |
Generates an image with control signal. |
Tensor<T> GenerateWithMultiControl(string prompt, Tensor<T>[] controlImages, ControlType[] controlTypes, double[] conditioningStrengths, string negativePrompt, int width, int height, int numInferenceSteps, double? guidanceScale, int? seed) |
Generates an image with multiple control signals. |