AutoencoderKL<T>
Models & Types · AiDotNet.Diffusion.VAE
The KL-regularized autoencoder of latent diffusion (Rombach et al., 2022), the VAE of Stable Diffusion.
Inheritance: VAEModelBase<T> → AutoencoderKL<T>
For Beginners
AutoencoderKL is the “image compressor” Stable Diffusion works through. It squeezes a 512x512 RGB image into a 64x64 grid of 4 latent channels; the diffusion model works there, and the decoder turns its result back into an image. The “KL” is the Kullback-Leibler penalty that keeps the latent close to a standard normal, so the latent space stays smooth.
How It Works
References:
- Paper: “High-Resolution Image Synthesis with Latent Diffusion Models” (Rombach et al., CVPR 2022), Sec. 3.1
and Appendix G
- Implementation: diffusers
AutoencoderKL(therunwayml/stable-diffusion-v1-5``vaefolder),
which this reproduces (tools/reference-data/autoencoder_kl_reference.py); LoadDiffusersWeights
loads its state dict.
Architecture: a 3x3 conv in; one stage of residual blocks per width, each stage but the last halving both axes with a
stride-2 conv; a middle block of residual block, single-head self-attention and residual block; GroupNorm, SiLU and a
3x3 conv to the posterior’s mean and log-variance; a 1x1 quant conv. The decoder mirrors it (a 1x1 post-quant conv
first, one more residual block per stage, nearest 2x upsampling then a conv). The widths are
baseChannels times each of channelMults.
Example
var vae = AutoencoderKL<float>.StableDiffusionV1();
var image = Tensor<float>.CreateRandom(new[] { 1, 3, 512, 512 });
var latent = vae.Encode(image, sampleMode: false); // [1, 4, 64, 64] for a [1, 3, 512, 512] image
var decoded = vae.Decode(latent);
Constructors
| Constructor | Summary |
|---|---|
AutoencoderKL(int inputChannels, int latentChannels, int baseChannels, int[] channelMults, int numResBlocks, int numGroups, double? latentScaleFactor, ILossFunction<T> lossFunction, int? seed) |
Initializes an AutoencoderKL; the defaults are Stable Diffusion v1.5’s (diffusers AutoencoderKL, runwayml/stable-diffusion-v1-5). |
Properties
| Property | Summary |
|---|---|
bool SupportsSlicing { get; } |
|
bool SupportsTiling { get; } |
|
double LatentScaleFactor { get; } |
|
int DownsampleFactor { get; } |
|
int InputChannels { get; } |
|
int LatentChannels { get; } |
Methods
| Method | Summary |
|---|---|
AutoencoderKL<T> Lightweight() |
A smaller VAE of the same architecture, for experiments. |
AutoencoderKL<T> SDXL() |
SDXL’s VAE: the same architecture with its own latent scale. |
AutoencoderKL<T> StableDiffusionV1() |
Stable Diffusion v1.5’s VAE. |
IReadOnlyCollection<string> LoadDiffusersWeights(Func<string, int[], double[]> read) |
Loads a diffusers AutoencoderKL state dict straight from source into the weights. |
T ComputeVAELoss(Tensor<T> image, Tensor<T> reconstruction, double klWeight) |
The VAE loss: reconstruction plus the KL divergence of the last encoding’s posterior, weighted. |
Tensor<T> Decode(Tensor<T> latent) |
|
Tensor<T> DecodeFromDiffusion(Tensor<T> latent) |
Decodes a diffusion latent (divided by LatentScaleFactor first) back to an image in [-1, 1]. |
Tensor<T> Encode(Tensor<T> image, bool sampleMode) |
|
Tensor<T> EncodeForDiffusion(Tensor<T> image, bool sampleMode) |
Encodes an image and scales the latent for diffusion. |
Tensor<T> Forward(Tensor<T> image) |
Encodes, samples the posterior and decodes. |
ValueTuple<Tensor<T>, Tensor<T>> EncodeWithDistribution(Tensor<T> image) |
|
void InvalidateCompiledHosts() |
Drops the decoder’s cached compiled plans, so the next Decode re-traces. |
void ResetState() |
Forgets the last encoding’s posterior. |
Fields
| Field | Summary |
|---|---|
double SdxlLatentScale |
The scaling factor of SDXL’s VAE. |
double StableDiffusionLatentScale |
The scaling factor of Stable Diffusion 1.x and 2.x’s VAE. |