StableDiffusion3Model<T>
Models & Types · AiDotNet.Diffusion.TextToImage
Stable Diffusion 3 / SD 3.5 model for text-to-image generation using rectified flow and MMDiT.
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → StableDiffusion3Model<T>
For Beginners
SD3 is the successor to SDXL with major architectural improvements:
How SD3 works:
- Text is encoded by THREE encoders: CLIP ViT-L/14, OpenCLIP ViT-bigG/14, and T5-XXL
- An MMDiT (Multi-Modal Diffusion Transformer) processes image and text tokens jointly
- Uses rectified flow instead of DDPM noise scheduling
- A new 16-channel VAE decodes latents to 1024x1024 images
Key characteristics:
- Triple text encoders (CLIP L + OpenCLIP G + T5-XXL)
- MMDiT: Joint attention between text and image tokens
- 16 latent channels (vs 4 in SD 1.5/SDXL)
- Rectified flow (linear noise schedule, v-prediction)
- SD3 Medium: 2B MMDiT parameters, 24 layers
- SD3.5 Large: 8B MMDiT parameters, 38 layers
- SD3.5 Large Turbo: 8B distilled for 4-step generation
Advantages:
- Superior text rendering in generated images
- Better prompt adherence than SDXL
- Higher quality details and compositions
- Scalable MMDiT architecture
Limitations:
- Higher compute requirements than SDXL
- Fewer community fine-tunes (newer ecosystem)
- T5-XXL encoder increases VRAM usage significantly
How It Works
Stable Diffusion 3 (SD3) is a next-generation text-to-image model by Stability AI that replaces the U-Net with a Multi-Modal Diffusion Transformer (MMDiT) and uses rectified flow matching instead of traditional DDPM-style noise schedules.
Technical specifications:
- Architecture: MMDiT (Multi-Modal Diffusion Transformer) + 16-channel VAE
- Text encoder 1: CLIP ViT-L/14 (768-dim)
- Text encoder 2: OpenCLIP ViT-bigG/14 (1280-dim)
- Text encoder 3: T5-XXL (4096-dim)
- Combined pooled embedding: 2048-dim (768 + 1280)
- Context dimension: 4096 (T5 embeddings for cross-attention)
- SD3 Medium: 2B params, hidden 1536, 24 layers, 24 heads
- SD3.5 Large: 8B params, hidden 2432, 38 layers, 38 heads
- VAE: 16 latent channels, scale factor 1.5305, shift 0.0609
- Training: Rectified flow matching with linear schedule
- Resolution: 1024x1024 (native)
Reference: Esser et al., “Scaling Rectified Flow Transformers for High-Resolution Image Synthesis”, ICML 2024
Example
// Create SD3 Medium with defaults
var sd3 = new StableDiffusion3Model<float>();
// Create SD3.5 Large
var sd35 = new StableDiffusion3Model<float>(variant: SD3Variant.Large);
// Generate an image
var image = sd3.GenerateFromText(
prompt: "A serene mountain landscape with the text 'Hello World' carved in stone",
width: 1024,
height: 1024,
numInferenceSteps: 28,
guidanceScale: 7.0,
seed: 42);
Constructors
| Constructor | Summary |
|---|---|
StableDiffusion3Model(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, MMDiTNoisePredictor<T> mmdit, StandardVAE<T> vae, IConditioningModule<T> conditioner, SD3Variant variant, int? seed) |
Initializes a new instance of StableDiffusion3Model with full customization support. |
Properties
| Property | Summary |
|---|---|
IConditioningModule<T> Conditioner { get; } |
|
INoisePredictor<T> NoisePredictor { get; } |
|
IVAEModel<T> VAE { get; } |
|
SD3Variant Variant { get; } |
Gets the model variant. |
int LatentChannels { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> GenerateFromText(string prompt, string negativePrompt, int width, int height, int numInferenceSteps, double? guidanceScale, int? seed) |
|
Tensor<T> ImageToImage(Tensor<T> inputImage, string prompt, string negativePrompt, double strength, int numInferenceSteps, double? guidanceScale, int? seed) |
Fields
| Field | Summary |
|---|---|
int DefaultHeight |
Default image height for SD3 (1024x1024). |
int DefaultWidth |
Default image width for SD3 (1024x1024). |