StableDiffusion2Model<T>

Models & Types · AiDotNet.Diffusion.TextToImage

Stable Diffusion 2.0/2.1 model for text-to-image generation.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → StableDiffusion2Model<T>

For Beginners

SD 2.0/2.1 is an upgraded version of SD 1.5 with key differences:

How SD 2.x works:

  1. Your text prompt is encoded by OpenCLIP ViT-H/14 (1024-dim embeddings)
  2. These embeddings guide a U-Net (865M parameters) that denoises in latent space
  3. A VAE decodes the denoised latent into a 768x768 image (SD 2.0) or 512/768 (SD 2.1)

Key differences from SD 1.5:

  • Text encoder: OpenCLIP ViT-H/14 (1024-dim) vs CLIP ViT-L/14 (768-dim)
  • Prediction type: v-prediction vs epsilon-prediction
  • Native resolution: 768x768 (SD 2.0) or 512x512/768x768 (SD 2.1)
  • Removed NSFW content from training data
  • Better at generating text in images

When to use SD 2.x:

  • Need v-prediction for certain workflows
  • Want OpenCLIP text encoder (different strengths than CLIP)
  • Need 768x768 native resolution

Limitations:

  • Smaller community ecosystem than SD 1.5
  • Some users prefer SD 1.5’s CLIP encoder for prompt adherence
  • Not as widely supported by third-party tools

How It Works

Stable Diffusion 2.x is the second generation of Stability AI’s text-to-image model. It uses OpenCLIP ViT-H/14 instead of the original CLIP ViT-L/14 text encoder, and introduces v-prediction as the default prediction type.

Technical specifications:

  • Architecture: U-Net with cross-attention and time embedding
  • Text encoder: OpenCLIP ViT-H/14 (632M parameters, 1024-dim, 77 max tokens)
  • U-Net: 865M parameters, base channels 320, [1, 2, 4, 4] multipliers
  • Cross-attention dimension: 1024 (matches OpenCLIP output)
  • VAE: KL-regularized autoencoder, 4 latent channels, scale factor 0.18215
  • Noise schedule: Scaled linear beta schedule, 1000 training timesteps
  • Prediction type: v-prediction (velocity prediction)

Reference: Rombach et al., “High-Resolution Image Synthesis with Latent Diffusion Models”, CVPR 2022

Example

// Create with industry-standard defaults
var sd2 = new StableDiffusion2Model<float>();
            
// Generate a 768x768 image from text
var image = sd2.GenerateFromText(
    prompt: "A photograph of a mountain landscape at golden hour",
    negativePrompt: "blurry, low quality, distorted",
    width: 768,
    height: 768,
    numInferenceSteps: 30,
    guidanceScale: 7.5,
    seed: 42);

Constructors

Constructor Summary
StableDiffusion2Model(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> unet, StandardVAE<T> vae, IConditioningModule<T> conditioner, bool useVPrediction, int? seed) Initializes a new instance of StableDiffusion2Model with full customization support.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> VAE { get; }
bool UsesVPrediction { get; } Gets whether this model uses v-prediction (velocity prediction).
int CrossAttentionDim { get; } Gets the cross-attention dimension (1024 for SD 2.x, matching OpenCLIP ViT-H/14).
int LatentChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> GenerateFromText(string prompt, string negativePrompt, int width, int height, int numInferenceSteps, double? guidanceScale, int? seed)
Tensor<T> ImageToImage(Tensor<T> inputImage, string prompt, string negativePrompt, double strength, int numInferenceSteps, double? guidanceScale, int? seed)

Fields

Field Summary
int DefaultHeight Default image height for SD 2.x generation.
int DefaultWidth Default image width for SD 2.x generation.