StableDiffusion15Model<T>

Models & Types · AiDotNet.Diffusion.TextToImage

Stable Diffusion 1.5 model for text-to-image generation.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → StableDiffusion15Model<T>

For Beginners

Stable Diffusion 1.5 is the “classic” text-to-image AI model.

How SD 1.5 works:

  1. Your text prompt is encoded by a CLIP ViT-L/14 text encoder into embeddings
  2. These embeddings guide a U-Net (865M parameters) that denoises in latent space
  3. A VAE decodes the denoised latent into a 512x512 image

Key characteristics:

  • Single text encoder: CLIP ViT-L/14 (768-dim embeddings)
  • U-Net: 865M parameters, channel multipliers [1, 2, 4, 4]
  • VAE: 4-channel latent space, 8x spatial downsampling
  • Native resolution: 512x512 pixels
  • Latent scale factor: 0.18215
  • Guidance scale: 7.5 (default)

When to use SD 1.5:

  • Huge community model ecosystem (thousands of fine-tunes available)
  • Lower resource requirements than SDXL (runs on 4GB+ VRAM)
  • Fast generation (20-50 steps)
  • Excellent for 512x512 generation

Limitations:

  • Lower resolution than SDXL (512x512 vs 1024x1024)
  • Single text encoder (less prompt understanding than dual-encoder models)
  • Occasional artifacts at hands, text, and complex scenes

How It Works

Stable Diffusion 1.5 (SD 1.5) is a latent diffusion model developed by Stability AI and Runway ML. It is the most widely used open-source text-to-image model and the foundation for an enormous ecosystem of fine-tunes, LoRAs, ControlNets, and community models.

Technical specifications:

  • Architecture: U-Net with cross-attention and time embedding
  • Text encoder: CLIP ViT-L/14 (63M parameters, 768-dim, 77 max tokens)
  • U-Net: 865M parameters, base channels 320, [1, 2, 4, 4] multipliers
  • Cross-attention dimension: 768 (matches CLIP output)
  • Attention resolutions: at 4x, 2x, and 1x downsampling levels
  • VAE: KL-regularized autoencoder, 4 latent channels, scale factor 0.18215
  • Noise schedule: Scaled linear beta schedule, 1000 training timesteps
  • Beta range: [0.00085, 0.012]
  • Prediction type: Epsilon (noise prediction)

Reference: Rombach et al., “High-Resolution Image Synthesis with Latent Diffusion Models”, CVPR 2022

Example

var maskTensor = Tensor<float>.CreateRandom(1, 3, 32, 32);
var existingImage = Tensor<float>.CreateRandom(1, 3, 32, 32);
// Create with industry-standard defaults
var sd15 = new StableDiffusion15Model<float>();
            
// Swap in your own components. Every parameter is optional, so supply only what you are replacing;
// a conditioner is omitted here because CLIPTextConditioner needs a tokenizer, which needs a
// vocabulary — worth its own example rather than three lines of setup in this one.
var customSd15 = new StableDiffusion15Model<float>(
    unet: new UNetNoisePredictor<float>(),
    vae: new StandardVAE<float>());
            
// Generate a 512x512 image from text
var image = sd15.GenerateFromText(
    prompt: "A photograph of a cat sitting on a windowsill at sunset",
    negativePrompt: "blurry, low quality, distorted",
    width: 512,
    height: 512,
    numInferenceSteps: 30,
    guidanceScale: 7.5,
    seed: 42);
            
// Image-to-image transformation
var transformed = sd15.ImageToImage(
    inputImage: existingImage,
    prompt: "Oil painting style, vibrant colors",
    strength: 0.7,
    numInferenceSteps: 30);
            
// Inpainting
var inpainted = sd15.Inpaint(
    inputImage: existingImage,
    mask: maskTensor,
    prompt: "A fluffy golden retriever",
    numInferenceSteps: 30);

Constructors

Constructor Summary
StableDiffusion15Model(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> unet, StandardVAE<T> vae, IConditioningModule<T> conditioner, int? seed) Initializes a new instance of StableDiffusion15Model with full customization support.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> VAE { get; }
int CrossAttentionDim { get; } Gets the cross-attention dimension (768 for SD 1.5, matching CLIP ViT-L/14).
int LatentChannels { get; }

Methods

Method Summary
List<Tensor<T>> GenerateVariations(string prompt, string negativePrompt, int count, int width, int height, int numInferenceSteps, double? guidanceScale, int? baseSeed) Generates multiple image variations from the same prompt using different seeds.
ModelMetadata<T> GetModelMetadata()
Tensor<T> GenerateFromText(string prompt, string negativePrompt, int width, int height, int numInferenceSteps, double? guidanceScale, int? seed) Generates an image from a text prompt using SD 1.5 defaults.
Tensor<T> ImageToImage(Tensor<T> inputImage, string prompt, string negativePrompt, double strength, int numInferenceSteps, double? guidanceScale, int? seed) Performs image-to-image transformation using SD 1.5.

Fields

Field Summary
int DefaultHeight Default image height for SD 1.5 generation.
int DefaultWidth Default image width for SD 1.5 generation.