DeepFloydIFModel<T>
Models & Types · AiDotNet.Diffusion.TextToImage
DeepFloyd IF model for cascaded text-to-image generation in pixel space.
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → DeepFloydIFModel<T>
For Beginners
DeepFloyd IF generates images by progressively upscaling:
How DeepFloyd IF works:
- Stage I: Generates a 64x64 pixel image from text (using T5-XXL embeddings)
- Stage II: Upscales 64x64 → 256x256 with text-guided super-resolution
- Stage III: Upscales 256x256 → 1024x1024 (optional, non-diffusion upscaler)
Key characteristics:
- Pixel-space diffusion (no VAE/latent space for Stages I and II)
- Text encoder: Frozen T5-XXL (4.7B parameters, 4096-dim embeddings)
- Stage I: ~900M parameters, 64x64 output
- Stage II: ~450M parameters, 256x256 output
- Stage III: Non-diffusion upscaler to 1024x1024
- Exceptional text rendering in images
Advantages:
- Best-in-class text rendering (can write legible text)
- Exceptional prompt adherence from T5-XXL
- Pixel-space avoids VAE artifacts
Limitations:
- Very large memory requirement (T5-XXL alone is ~10GB)
- Slower than latent diffusion models
- Multi-stage pipeline adds complexity
- Restricted license (not fully open-source)
How It Works
DeepFloyd IF is a three-stage cascaded diffusion model that operates in pixel space (not latent space), developed by DeepFloyd (Stability AI). It uses a frozen T5-XXL text encoder for exceptional text understanding and prompt adherence.
Technical specifications:
- Architecture: Cascaded pixel-space diffusion
- Stage I: U-Net, ~900M parameters, 64x64 output, 3 RGB channels
- Stage II: U-Net, ~450M parameters, 256x256 output, 6 channels (3 RGB + 3 low-res input)
- Text encoder: Frozen T5-XXL (4.7B params, 4096-dim, 256 max tokens)
- Noise schedule: Cosine beta schedule, 1000 training timesteps
- Prediction type: Epsilon prediction with dynamic thresholding
Reference: Saharia et al., “Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding”, 2022
Example
// Create with industry-standard defaults
var deepfloyd = new DeepFloydIFModel<float>();
// Generate a 64x64 base image (Stage I)
var image = deepfloyd.GenerateFromText(
prompt: "A sign that says 'Hello World' in neon lights",
negativePrompt: "blurry, low quality",
width: 64,
height: 64,
numInferenceSteps: 100,
guidanceScale: 7.0,
seed: 42);
Constructors
| Constructor | Summary |
|---|---|
DeepFloydIFModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> stageIUnet, UNetNoisePredictor<T> stageIIUnet, StandardVAE<T> vae, IConditioningModule<T> conditioner, bool useDynamicThresholding, int? seed) |
Initializes a new instance of DeepFloydIFModel with full customization support. |
Properties
| Property | Summary |
|---|---|
IConditioningModule<T> Conditioner { get; } |
|
INoisePredictor<T> NoisePredictor { get; } |
|
INoisePredictor<T> StageIINoisePredictor { get; } |
Gets the Stage II super-resolution noise predictor. |
IVAEModel<T> VAE { get; } |
|
bool UsesDynamicThresholding { get; } |
Gets whether dynamic thresholding is enabled. |
int CrossAttentionDim { get; } |
Gets the cross-attention dimension (4096 for T5-XXL). |
int LatentChannels { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> GenerateFromText(string prompt, string negativePrompt, int width, int height, int numInferenceSteps, double? guidanceScale, int? seed) |
|
Tensor<T> ImageToImage(Tensor<T> inputImage, string prompt, string negativePrompt, double strength, int numInferenceSteps, double? guidanceScale, int? seed) |
Fields
| Field | Summary |
|---|---|
int DefaultHeight |
Default image height for Stage I (64x64 base generation). |
int DefaultWidth |
Default image width for Stage I (64x64 base generation). |