UniControlNetModel<T>
Models & Types · AiDotNet.Diffusion.Control
Uni-ControlNet model for simultaneous multi-condition control with condition-specific adapters.
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → UniControlNetModel<T>
For Beginners
Uni-ControlNet combines multiple spatial conditions simultaneously.
How Uni-ControlNet works:
- Multiple condition images are provided (e.g., depth + edges + pose simultaneously)
- Each condition is processed by its specific adapter encoder
- Adapter features are mixed with learned weights for each condition
- Combined control features are injected into the SD 1.5 U-Net
- The U-Net generates an image guided by all conditions simultaneously
Key characteristics:
- Simultaneous multi-condition control (depth + edges + pose at same time)
- Condition-specific adapters with learned mixing weights
- Single forward pass for all conditions combined
- Compatible with SD 1.5 backbone
- No need to manually balance multiple ControlNets
When to use Uni-ControlNet:
- Multiple conditions applied simultaneously
- Complex spatial control combining depth, edges, and pose
- Single-model multi-condition pipeline
- When per-condition ControlNet loading is impractical
Limitations:
- SD 1.5 resolution (512x512)
- Adding new condition types requires retraining
- Condition interactions can produce unexpected results
- Not as flexible as separate ControlNets for per-condition tuning
How It Works
Uni-ControlNet allows composing multiple visual conditions simultaneously in a single forward pass. It uses condition-specific adapters that are mixed together with learned weights, enabling complex multi-condition spatial control.
Architecture components:
- SD 1.5 U-Net backbone (320 base channels, [1,2,4,4], 768-dim CLIP)
- Condition-specific encoder adapters for each control type
- Learned condition mixing weights for multi-condition composition
- Standard SD 1.5 VAE for image encoding/decoding
- DDIM scheduler for efficient inference
Technical specifications:
- Architecture: Multi-adapter ControlNet with condition mixing
- Backbone: SD 1.5 (320 base channels, [1,2,4,4] multipliers)
- Cross-attention: 768-dim (CLIP ViT-L/14)
- Adapters: condition-specific encoders with shared backbone
- Default resolution: 512x512
- Scheduler: DDIM
Reference: Zhao et al., “Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models”, NeurIPS 2023
Example
var uniControlNet = new UniControlNetModel<float>();
var image = uniControlNet.GenerateFromText(
prompt: "A photorealistic cityscape at sunset",
width: 512, height: 512,
numInferenceSteps: 20,
guidanceScale: 7.5);
Constructors
| Constructor | Summary |
|---|---|
UniControlNetModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> unet, StandardVAE<T> vae, IConditioningModule<T> conditioner, int? seed) |
Properties
| Property | Summary |
|---|---|
IConditioningModule<T> Conditioner { get; } |
|
INoisePredictor<T> NoisePredictor { get; } |
|
IVAEModel<T> VAE { get; } |
|
int LatentChannels { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> GenerateFromText(string prompt, string negativePrompt, int width, int height, int numInferenceSteps, double? guidanceScale, int? seed) |
Fields
| Field | Summary |
|---|---|
int DefaultHeight |
|
int DefaultWidth |