TemporalInterpolationVAE<T>

Models & Types · AiDotNet.Diffusion.VAE

Stable Video Diffusion’s VAE: the latent-diffusion KL encoder applied to each frame, and a decoder that mixes information across frames so the decoded clip is temporally consistent.

Inheritance: VAEModelBase<T> → TemporalInterpolationVAE<T>

Implements: IVideoVAEModel<T>

For Beginners

A video VAE compresses each frame to a small latent and back. Decoding every frame on its own makes the result flicker, so this VAE’s decoder also looks across neighbouring frames: each block blends a per-frame result with one that mixes the frames, and a final convolution over time smooths the clip.

How It Works

References:

  • Paper: “Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets” (Blattmann et al.,

2023), Sec. 3 and Appendix D (the temporal decoder)

  • Implementation: diffusers AutoencoderKLTemporalDecoder (the

stabilityai/stable-video-diffusion-img2vid``vae folder), which this reproduces (tools/reference-data/svd_vae_reference.py); LoadDiffusersWeights loads its state dict.

Architecture (diffusers’ AutoencoderKLTemporalDecoder): the encoder and its 1x1 quant conv are the KL autoencoder’s, run on every frame. The decoder is a 3x3 conv in; a middle block of spatio-temporal residual blocks around single-head self-attention; one stage per width from the deepest, of layersPerBlock + 1 spatio-temporal residual blocks and, but the last, a nearest 2x upsample; GroupNorm, SiLU and a 3x3 conv out; and a (3, 1, 1) convolution over time. Each spatio-temporal block is a 2D residual block on every frame, a residual block of (3, 1, 1) convolutions across the frames, and a learned blend of the two. There is no post-quant conv. A clip is [batch, channels, frames, height, width]; an image, [batch, channels, height, width], is a one-frame clip.

Example

var vae = new TemporalInterpolationVAE<float>();
var clip = Tensor<float>.CreateRandom(new[] { 1, 3, 14, 576, 1024 });
var latent = vae.Encode(clip, sampleMode: false); // [1, 4, 14, 72, 128] for a [1, 3, 14, 576, 1024] clip
var decoded = vae.Decode(latent);

Constructors

Constructor Summary
TemporalInterpolationVAE(int inputChannels, int latentChannels, int[] blockOutChannels, int layersPerBlock, int normGroups, double latentScaleFactor, ILossFunction<T> lossFunction, int? seed) Initializes the VAE; the defaults are Stable Video Diffusion’s (diffusers AutoencoderKLTemporalDecoder, stabilityai/stable-video-diffusion-img2vid).

Properties

Property Summary
double LatentScaleFactor { get; }
int DownsampleFactor { get; }
int InputChannels { get; }
int LatentChannels { get; }
int TemporalCompression { get; }

Methods

Method Summary
IReadOnlyCollection<string> LoadDiffusersWeights(Func<string, int[], double[]> read) Loads a diffusers AutoencoderKLTemporalDecoder state dict straight from source into the weights.
Tensor<T> Decode(Tensor<T> latent)
Tensor<T> Encode(Tensor<T> input, bool sampleMode)
ValueTuple<Tensor<T>, Tensor<T>> EncodeWithDistribution(Tensor<T> input)
int GetLatentFrameCount(int frames)

Fields

Field Summary
double SvdLatentScale The scaling factor of Stable Video Diffusion’s VAE.