DallE2Model<T>
Models & Types · AiDotNet.Diffusion.TextToImage
DALL-E 2 (unCLIP) model for text-to-image generation via CLIP-guided diffusion.
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → DallE2Model<T>
For Beginners
DALL-E 2 generates images through CLIP space:
How DALL-E 2 works:
- Text is encoded by CLIP ViT-L/14 text encoder (768-dim)
- A diffusion prior maps text embeddings → CLIP image embeddings
- A diffusion decoder generates 64x64 images from image embeddings
- Two upsampler stages scale to 256x256 → 1024x1024
Key characteristics:
- Two-stage: Diffusion prior + Diffusion decoder
- CLIP-guided: operates in CLIP embedding space
- Text encoder: CLIP ViT-L/14 (768-dim)
- Prior: Diffusion transformer mapping text→image embeddings
- Decoder: Modified GLIDE model, pixel-space diffusion
- Supports image variations (re-generate from CLIP embedding)
Advantages:
- Natural image variations through CLIP space manipulation
- Good compositional understanding
- Supports text-guided image editing
Limitations:
- Not open-source (proprietary to OpenAI)
- Superseded by DALL-E 3 in quality
- Sometimes struggles with text rendering
How It Works
DALL-E 2 is a text-to-image model developed by OpenAI that uses a two-stage pipeline: a prior that generates CLIP image embeddings from text, and a decoder that generates images from those embeddings. This approach is also known as “unCLIP”.
Technical specifications:
- Architecture: Diffusion Prior + Diffusion Decoder (unCLIP)
- Prior: Transformer-based diffusion, maps CLIP text→image embeddings
- Decoder: Modified GLIDE U-Net, ~3.5B parameters, 64x64 base
- Text encoder: CLIP ViT-L/14 (768-dim embeddings)
- Image encoder: CLIP ViT-L/14 (768-dim, used for prior training)
- Noise schedule: Linear beta, 1000 training timesteps
- Upsampler: Two ADM upsampler stages (64→256→1024)
Reference: Ramesh et al., “Hierarchical Text-Conditional Image Generation with CLIP Latents”, 2022
Example
// Create with industry-standard defaults
var dalle2 = new DallE2Model<float>();
// Generate a 64x64 base image
var image = dalle2.GenerateFromText(
prompt: "An astronaut riding a horse in the style of Andy Warhol",
negativePrompt: null,
width: 64,
height: 64,
numInferenceSteps: 64,
guidanceScale: 4.0,
seed: 42);
Constructors
| Constructor | Summary |
|---|---|
DallE2Model(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> priorUnet, UNetNoisePredictor<T> decoderUnet, StandardVAE<T> vae, IConditioningModule<T> conditioner, int? seed) |
Initializes a new instance of DallE2Model with full customization support. |
Properties
| Property | Summary |
|---|---|
IConditioningModule<T> Conditioner { get; } |
|
INoisePredictor<T> DiffusionPrior { get; } |
Gets the diffusion prior that maps text embeddings to CLIP image embeddings. |
INoisePredictor<T> NoisePredictor { get; } |
|
IVAEModel<T> VAE { get; } |
|
int ClipDimension { get; } |
Gets the CLIP embedding dimension (768). |
int LatentChannels { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> GenerateFromText(string prompt, string negativePrompt, int width, int height, int numInferenceSteps, double? guidanceScale, int? seed) |
|
Tensor<T> ImageToImage(Tensor<T> inputImage, string prompt, string negativePrompt, double strength, int numInferenceSteps, double? guidanceScale, int? seed) |
Fields
| Field | Summary |
|---|---|
int DefaultHeight |
Default image height for DALL-E 2 base decoder (64x64). |
int DefaultWidth |
Default image width for DALL-E 2 base decoder (64x64). |