DeepFloydIFModel<T>

Models & Types · AiDotNet.Diffusion.TextToImage

DeepFloyd IF model for cascaded text-to-image generation in pixel space.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → DeepFloydIFModel<T>

For Beginners

DeepFloyd IF generates images by progressively upscaling:

How DeepFloyd IF works:

  1. Stage I: Generates a 64x64 pixel image from text (using T5-XXL embeddings)
  2. Stage II: Upscales 64x64 → 256x256 with text-guided super-resolution
  3. Stage III: Upscales 256x256 → 1024x1024 (optional, non-diffusion upscaler)

Key characteristics:

  • Pixel-space diffusion (no VAE/latent space for Stages I and II)
  • Text encoder: Frozen T5-XXL (4.7B parameters, 4096-dim embeddings)
  • Stage I: ~900M parameters, 64x64 output
  • Stage II: ~450M parameters, 256x256 output
  • Stage III: Non-diffusion upscaler to 1024x1024
  • Exceptional text rendering in images

Advantages:

  • Best-in-class text rendering (can write legible text)
  • Exceptional prompt adherence from T5-XXL
  • Pixel-space avoids VAE artifacts

Limitations:

  • Very large memory requirement (T5-XXL alone is ~10GB)
  • Slower than latent diffusion models
  • Multi-stage pipeline adds complexity
  • Restricted license (not fully open-source)

How It Works

DeepFloyd IF is a three-stage cascaded diffusion model that operates in pixel space (not latent space), developed by DeepFloyd (Stability AI). It uses a frozen T5-XXL text encoder for exceptional text understanding and prompt adherence.

Technical specifications:

  • Architecture: Cascaded pixel-space diffusion
  • Stage I: U-Net, ~900M parameters, 64x64 output, 3 RGB channels
  • Stage II: U-Net, ~450M parameters, 256x256 output, 6 channels (3 RGB + 3 low-res input)
  • Text encoder: Frozen T5-XXL (4.7B params, 4096-dim, 256 max tokens)
  • Noise schedule: Cosine beta schedule, 1000 training timesteps
  • Prediction type: Epsilon prediction with dynamic thresholding

Reference: Saharia et al., “Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding”, 2022

Example

// Create with industry-standard defaults
var deepfloyd = new DeepFloydIFModel<float>();
            
// Generate a 64x64 base image (Stage I)
var image = deepfloyd.GenerateFromText(
    prompt: "A sign that says 'Hello World' in neon lights",
    negativePrompt: "blurry, low quality",
    width: 64,
    height: 64,
    numInferenceSteps: 100,
    guidanceScale: 7.0,
    seed: 42);

Constructors

Constructor Summary
DeepFloydIFModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> stageIUnet, UNetNoisePredictor<T> stageIIUnet, StandardVAE<T> vae, IConditioningModule<T> conditioner, bool useDynamicThresholding, int? seed) Initializes a new instance of DeepFloydIFModel with full customization support.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
INoisePredictor<T> StageIINoisePredictor { get; } Gets the Stage II super-resolution noise predictor.
IVAEModel<T> VAE { get; }
bool UsesDynamicThresholding { get; } Gets whether dynamic thresholding is enabled.
int CrossAttentionDim { get; } Gets the cross-attention dimension (4096 for T5-XXL).
int LatentChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> GenerateFromText(string prompt, string negativePrompt, int width, int height, int numInferenceSteps, double? guidanceScale, int? seed)
Tensor<T> ImageToImage(Tensor<T> inputImage, string prompt, string negativePrompt, double strength, int numInferenceSteps, double? guidanceScale, int? seed)

Fields

Field Summary
int DefaultHeight Default image height for Stage I (64x64 base generation).
int DefaultWidth Default image width for Stage I (64x64 base generation).