T2IAdapterModel<T>

Models & Types · AiDotNet.Diffusion.Control

T2I-Adapter model for adding spatial control to text-to-image diffusion models.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → T2IAdapterModel<T>

For Beginners

T2I-Adapter adds structural guidance to image generation:

How T2I-Adapter works:

  1. A spatial condition (depth map, sketch, pose, etc.) is processed by the adapter network
  2. The adapter produces multi-scale feature maps matching the U-Net encoder stages
  3. These features are added to the U-Net during denoising for structural guidance
  4. The base SD model generates images following both text AND spatial conditions

Key characteristics:

  • Lightweight: Only ~77M parameters (vs ~860M for ControlNet)
  • Pluggable: Works with any SD 1.5 or SDXL base model
  • Composable: Multiple adapters can be combined (e.g., depth + sketch)
  • No base model modification: Adapter weights are separate
  • Fast training: Much faster to train than ControlNet

Supported condition types:

  • Depth maps (MiDaS, ZoeDepth)
  • Canny edge maps
  • Sketch/line art
  • OpenPose skeleton
  • Semantic segmentation
  • Color palette

Advantages over ControlNet:

  • 10x fewer parameters
  • Faster training and inference
  • Composable (stack multiple adapters)
  • Smaller model files

Limitations:

  • Less precise control than ControlNet
  • May not preserve fine spatial details as well

How It Works

T2I-Adapter is a lightweight adapter architecture that adds spatial conditioning (depth maps, sketches, pose, segmentation, etc.) to pre-trained text-to-image models without modifying the base model weights.

Technical specifications:

  • Architecture: Lightweight encoder adapter with multi-scale feature injection
  • Adapter: ~77M parameters, 4 downsampling stages matching U-Net encoder
  • Compatible base models: SD 1.5 (768-dim), SDXL (varies)
  • Input conditions: Any spatial map (depth, edge, pose, etc.)
  • Adapter scale: Adjustable strength [0.0, 1.0] for blending
  • Adapter channels: [320, 640, 1280, 1280] matching SD U-Net

Reference: Mou et al., “T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models”, AAAI 2024

Example

// Create T2I-Adapter with SD 1.5 as base
var adapter = new T2IAdapterModel<float>();
            
// Generate a 512x512 image with depth-guided control
var image = adapter.GenerateFromText(
    prompt: "A house on a hill at sunset",
    negativePrompt: "blurry, low quality",
    width: 512,
    height: 512,
    numInferenceSteps: 30,
    guidanceScale: 7.5,
    seed: 42);

Constructors

Constructor Summary
T2IAdapterModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> unet, UNetNoisePredictor<T> adapterNetwork, StandardVAE<T> vae, IConditioningModule<T> conditioner, double adapterScale, int? seed) Initializes a new instance of T2IAdapterModel with full customization support.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> AdapterNetwork { get; } Gets the adapter network that processes spatial conditions.
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> VAE { get; }
double AdapterScale { get; } Gets the adapter conditioning scale [0.0, 1.0].
int LatentChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> GenerateFromText(string prompt, string negativePrompt, int width, int height, int numInferenceSteps, double? guidanceScale, int? seed)
Tensor<T> ImageToImage(Tensor<T> inputImage, string prompt, string negativePrompt, double strength, int numInferenceSteps, double? guidanceScale, int? seed)

Fields

Field Summary
int DefaultHeight Default image height for T2I-Adapter (matches SD 1.5).
int DefaultWidth Default image width for T2I-Adapter (matches SD 1.5).