DallE2Model<T>

Models & Types · AiDotNet.Diffusion.TextToImage

DALL-E 2 (unCLIP) model for text-to-image generation via CLIP-guided diffusion.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → DallE2Model<T>

For Beginners

DALL-E 2 generates images through CLIP space:

How DALL-E 2 works:

  1. Text is encoded by CLIP ViT-L/14 text encoder (768-dim)
  2. A diffusion prior maps text embeddings → CLIP image embeddings
  3. A diffusion decoder generates 64x64 images from image embeddings
  4. Two upsampler stages scale to 256x256 → 1024x1024

Key characteristics:

  • Two-stage: Diffusion prior + Diffusion decoder
  • CLIP-guided: operates in CLIP embedding space
  • Text encoder: CLIP ViT-L/14 (768-dim)
  • Prior: Diffusion transformer mapping text→image embeddings
  • Decoder: Modified GLIDE model, pixel-space diffusion
  • Supports image variations (re-generate from CLIP embedding)

Advantages:

  • Natural image variations through CLIP space manipulation
  • Good compositional understanding
  • Supports text-guided image editing

Limitations:

  • Not open-source (proprietary to OpenAI)
  • Superseded by DALL-E 3 in quality
  • Sometimes struggles with text rendering

How It Works

DALL-E 2 is a text-to-image model developed by OpenAI that uses a two-stage pipeline: a prior that generates CLIP image embeddings from text, and a decoder that generates images from those embeddings. This approach is also known as “unCLIP”.

Technical specifications:

  • Architecture: Diffusion Prior + Diffusion Decoder (unCLIP)
  • Prior: Transformer-based diffusion, maps CLIP text→image embeddings
  • Decoder: Modified GLIDE U-Net, ~3.5B parameters, 64x64 base
  • Text encoder: CLIP ViT-L/14 (768-dim embeddings)
  • Image encoder: CLIP ViT-L/14 (768-dim, used for prior training)
  • Noise schedule: Linear beta, 1000 training timesteps
  • Upsampler: Two ADM upsampler stages (64→256→1024)

Reference: Ramesh et al., “Hierarchical Text-Conditional Image Generation with CLIP Latents”, 2022

Example

// Create with industry-standard defaults
var dalle2 = new DallE2Model<float>();
            
// Generate a 64x64 base image
var image = dalle2.GenerateFromText(
    prompt: "An astronaut riding a horse in the style of Andy Warhol",
    negativePrompt: null,
    width: 64,
    height: 64,
    numInferenceSteps: 64,
    guidanceScale: 4.0,
    seed: 42);

Constructors

Constructor Summary
DallE2Model(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> priorUnet, UNetNoisePredictor<T> decoderUnet, StandardVAE<T> vae, IConditioningModule<T> conditioner, int? seed) Initializes a new instance of DallE2Model with full customization support.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> DiffusionPrior { get; } Gets the diffusion prior that maps text embeddings to CLIP image embeddings.
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> VAE { get; }
int ClipDimension { get; } Gets the CLIP embedding dimension (768).
int LatentChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> GenerateFromText(string prompt, string negativePrompt, int width, int height, int numInferenceSteps, double? guidanceScale, int? seed)
Tensor<T> ImageToImage(Tensor<T> inputImage, string prompt, string negativePrompt, double strength, int numInferenceSteps, double? guidanceScale, int? seed)

Fields

Field Summary
int DefaultHeight Default image height for DALL-E 2 base decoder (64x64).
int DefaultWidth Default image width for DALL-E 2 base decoder (64x64).