TemporalInterpolationVAE<T>
Models & Types · AiDotNet.Diffusion.VAE
Stable Video Diffusion’s VAE: the latent-diffusion KL encoder applied to each frame, and a decoder that mixes information across frames so the decoded clip is temporally consistent.
Inheritance: VAEModelBase<T> → TemporalInterpolationVAE<T>
Implements: IVideoVAEModel<T>
For Beginners
A video VAE compresses each frame to a small latent and back. Decoding every frame on its own makes the result flicker, so this VAE’s decoder also looks across neighbouring frames: each block blends a per-frame result with one that mixes the frames, and a final convolution over time smooths the clip.
How It Works
References:
- Paper: “Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets” (Blattmann et al.,
2023), Sec. 3 and Appendix D (the temporal decoder)
- Implementation: diffusers
AutoencoderKLTemporalDecoder(the
stabilityai/stable-video-diffusion-img2vid``vae folder), which this reproduces
(tools/reference-data/svd_vae_reference.py); LoadDiffusersWeights loads its state dict.
Architecture (diffusers’ AutoencoderKLTemporalDecoder): the encoder and its 1x1 quant conv are the KL
autoencoder’s, run on every frame. The decoder is a 3x3 conv in; a middle block of spatio-temporal residual blocks
around single-head self-attention; one stage per width from the deepest, of layersPerBlock + 1
spatio-temporal residual blocks and, but the last, a nearest 2x upsample; GroupNorm, SiLU and a 3x3 conv out; and a
(3, 1, 1) convolution over time. Each spatio-temporal block is a 2D residual block on every frame, a residual block
of (3, 1, 1) convolutions across the frames, and a learned blend of the two. There is no post-quant conv. A clip is
[batch, channels, frames, height, width]; an image, [batch, channels, height, width], is a one-frame clip.
Example
var vae = new TemporalInterpolationVAE<float>();
var clip = Tensor<float>.CreateRandom(new[] { 1, 3, 14, 576, 1024 });
var latent = vae.Encode(clip, sampleMode: false); // [1, 4, 14, 72, 128] for a [1, 3, 14, 576, 1024] clip
var decoded = vae.Decode(latent);
Constructors
| Constructor | Summary |
|---|---|
TemporalInterpolationVAE(int inputChannels, int latentChannels, int[] blockOutChannels, int layersPerBlock, int normGroups, double latentScaleFactor, ILossFunction<T> lossFunction, int? seed) |
Initializes the VAE; the defaults are Stable Video Diffusion’s (diffusers AutoencoderKLTemporalDecoder, stabilityai/stable-video-diffusion-img2vid). |
Properties
| Property | Summary |
|---|---|
double LatentScaleFactor { get; } |
|
int DownsampleFactor { get; } |
|
int InputChannels { get; } |
|
int LatentChannels { get; } |
|
int TemporalCompression { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyCollection<string> LoadDiffusersWeights(Func<string, int[], double[]> read) |
Loads a diffusers AutoencoderKLTemporalDecoder state dict straight from source into the weights. |
Tensor<T> Decode(Tensor<T> latent) |
|
Tensor<T> Encode(Tensor<T> input, bool sampleMode) |
|
ValueTuple<Tensor<T>, Tensor<T>> EncodeWithDistribution(Tensor<T> input) |
|
int GetLatentFrameCount(int frames) |
Fields
| Field | Summary |
|---|---|
double SvdLatentScale |
The scaling factor of Stable Video Diffusion’s VAE. |