ImagenModel<T>

Models & Types · AiDotNet.Diffusion.TextToImage

Imagen model for cascaded text-to-image generation with T5 text encoding.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → ImagenModel<T>

For Beginners

Imagen generates images by starting small and upscaling:

How Imagen works:

  1. Text is encoded by a frozen T5-XXL language model (4096-dim embeddings)
  2. A base diffusion model generates a 64x64 image from the text embeddings
  3. A first super-resolution model upscales 64x64 → 256x256
  4. A second super-resolution model upscales 256x256 → 1024x1024

Key characteristics:

  • Cascaded pixel-space diffusion: 64→256→1024
  • Text encoder: Frozen T5-XXL (4.7B parameters, 4096-dim)
  • Base model: ~2B parameters, 64x64 output
  • Super-res 1: ~600M parameters, 256x256 output
  • Super-res 2: ~400M parameters, 1024x1024 output
  • Uses Efficient U-Net architecture
  • Dynamic thresholding for improved image quality

Key innovations:

  • Demonstrated that text encoder quality is most important
  • Introduced dynamic thresholding (better high guidance scales)
  • Efficient U-Net: memory-efficient attention, shifted convolutions
  • Noise conditioning augmentation for super-resolution stages

Limitations:

  • Not open-source (proprietary to Google)
  • Very large compute requirements
  • Three separate models needed for full pipeline

How It Works

Imagen is a cascaded text-to-image diffusion model developed by Google Brain. It demonstrates that large frozen language models (T5-XXL) are highly effective for text-to-image generation, and that scaling the text encoder matters more than scaling the image model.

Technical specifications:

  • Architecture: Cascaded pixel-space diffusion with Efficient U-Net
  • Base model: 64x64, ~2B parameters, 3 RGB channels
  • Super-res 1: 64→256, ~600M parameters
  • Super-res 2: 256→1024, ~400M parameters
  • Text encoder: Frozen T5-XXL (4.7B params, 4096-dim, 256 max tokens)
  • Noise schedule: Cosine schedule, 1000 training timesteps
  • Prediction type: Epsilon prediction with dynamic thresholding
  • Noise conditioning augmentation for super-res stages

Reference: Saharia et al., “Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding”, NeurIPS 2022

Example

// Create with industry-standard defaults
var imagen = new ImagenModel<float>();
            
// Generate a 64x64 base image
var image = imagen.GenerateFromText(
    prompt: "A brain riding a rocketship heading towards the moon",
    negativePrompt: null,
    width: 64,
    height: 64,
    numInferenceSteps: 100,
    guidanceScale: 7.5,
    seed: 42);

Constructors

Constructor Summary
ImagenModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> baseUnet, UNetNoisePredictor<T> superRes1Unet, StandardVAE<T> vae, IConditioningModule<T> conditioner, double dynamicThresholdPercentile, int? seed) Initializes a new instance of ImagenModel with full customization support.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
INoisePredictor<T> SuperResolution1 { get; } Gets the super-resolution Stage 1 noise predictor (64→256).
IVAEModel<T> VAE { get; }
double DynamicThresholdPercentile { get; } Gets the dynamic thresholding percentile.
int CrossAttentionDim { get; } Gets the cross-attention dimension (4096 for T5-XXL).
int LatentChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> GenerateFromText(string prompt, string negativePrompt, int width, int height, int numInferenceSteps, double? guidanceScale, int? seed)
Tensor<T> ImageToImage(Tensor<T> inputImage, string prompt, string negativePrompt, double strength, int numInferenceSteps, double? guidanceScale, int? seed)

Fields

Field Summary
int DefaultHeight Default image height for Imagen base model (64x64).
int DefaultWidth Default image width for Imagen base model (64x64).