AutoencoderKL<T>

Models & Types · AiDotNet.Diffusion.VAE

The KL-regularized autoencoder of latent diffusion (Rombach et al., 2022), the VAE of Stable Diffusion.

Inheritance: VAEModelBase<T> → AutoencoderKL<T>

For Beginners

AutoencoderKL is the “image compressor” Stable Diffusion works through. It squeezes a 512x512 RGB image into a 64x64 grid of 4 latent channels; the diffusion model works there, and the decoder turns its result back into an image. The “KL” is the Kullback-Leibler penalty that keeps the latent close to a standard normal, so the latent space stays smooth.

How It Works

References:

  • Paper: “High-Resolution Image Synthesis with Latent Diffusion Models” (Rombach et al., CVPR 2022), Sec. 3.1

and Appendix G

  • Implementation: diffusers AutoencoderKL (the runwayml/stable-diffusion-v1-5``vae folder),

which this reproduces (tools/reference-data/autoencoder_kl_reference.py); LoadDiffusersWeights loads its state dict.

Architecture: a 3x3 conv in; one stage of residual blocks per width, each stage but the last halving both axes with a stride-2 conv; a middle block of residual block, single-head self-attention and residual block; GroupNorm, SiLU and a 3x3 conv to the posterior’s mean and log-variance; a 1x1 quant conv. The decoder mirrors it (a 1x1 post-quant conv first, one more residual block per stage, nearest 2x upsampling then a conv). The widths are baseChannels times each of channelMults.

Example

var vae = AutoencoderKL<float>.StableDiffusionV1();
var image = Tensor<float>.CreateRandom(new[] { 1, 3, 512, 512 });
var latent = vae.Encode(image, sampleMode: false); // [1, 4, 64, 64] for a [1, 3, 512, 512] image
var decoded = vae.Decode(latent);

Constructors

Constructor Summary
AutoencoderKL(int inputChannels, int latentChannels, int baseChannels, int[] channelMults, int numResBlocks, int numGroups, double? latentScaleFactor, ILossFunction<T> lossFunction, int? seed) Initializes an AutoencoderKL; the defaults are Stable Diffusion v1.5’s (diffusers AutoencoderKL, runwayml/stable-diffusion-v1-5).

Properties

Property Summary
bool SupportsSlicing { get; }
bool SupportsTiling { get; }
double LatentScaleFactor { get; }
int DownsampleFactor { get; }
int InputChannels { get; }
int LatentChannels { get; }

Methods

Method Summary
AutoencoderKL<T> Lightweight() A smaller VAE of the same architecture, for experiments.
AutoencoderKL<T> SDXL() SDXL’s VAE: the same architecture with its own latent scale.
AutoencoderKL<T> StableDiffusionV1() Stable Diffusion v1.5’s VAE.
IReadOnlyCollection<string> LoadDiffusersWeights(Func<string, int[], double[]> read) Loads a diffusers AutoencoderKL state dict straight from source into the weights.
T ComputeVAELoss(Tensor<T> image, Tensor<T> reconstruction, double klWeight) The VAE loss: reconstruction plus the KL divergence of the last encoding’s posterior, weighted.
Tensor<T> Decode(Tensor<T> latent)
Tensor<T> DecodeFromDiffusion(Tensor<T> latent) Decodes a diffusion latent (divided by LatentScaleFactor first) back to an image in [-1, 1].
Tensor<T> Encode(Tensor<T> image, bool sampleMode)
Tensor<T> EncodeForDiffusion(Tensor<T> image, bool sampleMode) Encodes an image and scales the latent for diffusion.
Tensor<T> Forward(Tensor<T> image) Encodes, samples the posterior and decodes.
ValueTuple<Tensor<T>, Tensor<T>> EncodeWithDistribution(Tensor<T> image)
void InvalidateCompiledHosts() Drops the decoder’s cached compiled plans, so the next Decode re-traces.
void ResetState() Forgets the last encoding’s posterior.

Fields

Field Summary
double SdxlLatentScale The scaling factor of SDXL’s VAE.
double StableDiffusionLatentScale The scaling factor of Stable Diffusion 1.x and 2.x’s VAE.