IVAEModel<T>

Interfaces · AiDotNet.Interfaces

Interface for Variational Autoencoder (VAE) models used in latent diffusion.

Implements: IFeatureAware, IFullModel<T, Tensor<T>, Tensor<T>>, IGradientComputable<T, Tensor<T>, Tensor<T>>

Known implementations: IVideoVAEModel<T>, VAEModelBase<T>

For Beginners

A VAE is like a very smart image compressor and decompressor.

How it works:

  1. Encoder: Takes a full-size image (e.g., 512x512x3) and compresses it to a small latent (e.g., 64x64x4)
  2. Decoder: Takes the small latent and reconstructs a full-size image
  3. The compression is lossy but learned to preserve important visual information

Why use a VAE in diffusion?

  • Full images are huge (512x512x3 = 786,432 values)
  • Latents are small (64x64x4 = 16,384 values) - 48x smaller!
  • Diffusion in latent space is much faster
  • Quality remains high because the VAE learns what matters

Different VAE types:

  • Standard VAE: Original Stable Diffusion VAE, 4 latent channels
  • Tiny VAE: Faster but lower quality, good for previews
  • Temporal VAE: Video-aware VAE that handles frame consistency

How It Works

VAEs are used in latent diffusion models to compress images into a lower-dimensional latent space where the diffusion process operates. This makes training and generation much more efficient than operating in pixel space.

This interface extends IFullModel to provide all standard model capabilities (training, saving, loading, gradients, checkpointing, etc.).

Properties

Property Summary
bool SupportsSlicing { get; } Gets whether this VAE uses slicing for sequential processing.
bool SupportsTiling { get; } Gets whether this VAE uses tiling for memory-efficient encoding/decoding.
double LatentScaleFactor { get; } Gets the scale factor for latent values.
int DownsampleFactor { get; } Gets the spatial downsampling factor.
int InputChannels { get; } Gets the number of input channels (image channels).
int LatentChannels { get; } Gets the number of latent channels.

Methods

Method Summary
Tensor<T> Decode(Tensor<T> latent) Decodes a latent representation back to image space.
Tensor<T> Encode(Tensor<T> image, bool sampleMode) Encodes an image into the latent space.
Tensor<T> Sample(Tensor<T> mean, Tensor<T> logVariance, int? seed) Samples from the latent distribution using the reparameterization trick.
Tensor<T> ScaleLatent(Tensor<T> latent) Scales latent values for use in diffusion (applies LatentScaleFactor).
Tensor<T> UnscaleLatent(Tensor<T> latent) Unscales latent values before decoding (inverts LatentScaleFactor).
ValueTuple<Tensor<T>, Tensor<T>> EncodeWithDistribution(Tensor<T> image) Encodes and returns both mean and log variance (for training).
void SetSlicingEnabled(bool enabled) Enables or disables slicing mode.
void SetTilingEnabled(bool enabled) Enables or disables tiling mode.