ILatentDiffusionModel<T>

Interfaces · AiDotNet.Interfaces

Interface for latent diffusion models that operate in a compressed latent space.

Implements: IDiffusionModel<T>

Known implementations: LatentDiffusionModelBase<T>

For Beginners

Latent diffusion combines the power of diffusion models with the efficiency of autoencoders.

How it works:

  1. A VAE compresses images (512x512) into small latents (64x64)
  2. Diffusion happens in this compressed space (much faster!)
  3. The VAE decompresses the result back to a full image

Benefits:

  • Training is ~50x faster than pixel-space diffusion
  • Generation is ~50x faster
  • Quality remains very high
  • Enables practical high-resolution generation

Key components:

  • VAE: Compresses and decompresses images
  • Noise Predictor (U-Net/DiT): Predicts noise in latent space
  • Scheduler: Controls the denoising process
  • Conditioner: Encodes text/images for guided generation

How It Works

Latent diffusion models are a highly efficient variant of diffusion models that perform the denoising process in a compressed latent space rather than pixel space. This is the architecture behind Stable Diffusion and many other state-of-the-art generative models.

This interface extends IDiffusionModel with latent-space specific operations.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; } Gets the conditioning module (optional, for conditioned generation).
INoisePredictor<T> NoisePredictor { get; } Gets the noise predictor model (U-Net, DiT, etc.).
IVAEModel<T> VAE { get; } Gets the VAE model used for encoding and decoding.
bool SupportsInpainting { get; } Gets whether this model supports inpainting.
bool SupportsNegativePrompt { get; } Gets whether this model supports negative prompts.
double GuidanceScale { get; } Gets the default guidance scale for classifier-free guidance.
int LatentChannels { get; } Gets the number of latent channels.

Methods

Method Summary
Tensor<T> DecodeFromLatent(Tensor<T> latent) Decodes a latent representation back to an image.
Tensor<T> EncodeToLatent(Tensor<T> image, bool sampleMode) Encodes an image into latent space.
Tensor<T> GenerateFromText(string prompt, string negativePrompt, int width, int height, int numInferenceSteps, double? guidanceScale, int? seed) Generates images from text prompts using classifier-free guidance.
Tensor<T> ImageToImage(Tensor<T> inputImage, string prompt, string negativePrompt, double strength, int numInferenceSteps, double? guidanceScale, int? seed) Performs image-to-image generation (style transfer, editing).
Tensor<T> Inpaint(Tensor<T> inputImage, Tensor<T> mask, string prompt, string negativePrompt, int numInferenceSteps, double? guidanceScale, int? seed) Performs inpainting (filling in masked regions).
void SetGuidanceScale(double scale) Sets the guidance scale for classifier-free guidance.