TemporalVAE<T>

Models & Types · AiDotNet.Diffusion.VAE

Temporal-aware Variational Autoencoder for video diffusion models.

Inheritance: VAEModelBase<T> → TemporalVAE<T>

For Beginners

While a standard VAE processes each frame independently, TemporalVAE considers relationships between consecutive frames:

Standard VAE approach (per-frame):

  • Frame 1 -> Latent 1 (no knowledge of other frames)
  • Frame 2 -> Latent 2 (no knowledge of other frames)
  • Result: Possible flickering/inconsistency between frames

TemporalVAE approach:

  • Frames 1,2,3,… -> Encode with temporal awareness
  • Latent knows about neighboring frames
  • Result: Smoother, more consistent video

Key features:

  • 3D convolutions that span across time dimension
  • Temporal attention for long-range frame relationships
  • Optional causal mode for streaming/autoregressive generation

Used in: Stable Video Diffusion, Video LDM, and similar models.

How It Works

The TemporalVAE extends the standard VAE to handle video data by incorporating temporal awareness into the encoding and decoding process. This helps maintain temporal consistency across frames when used in video diffusion models.

Architecture details:

  • Input: [batch, channels, frames, height, width] video tensor
  • Encoder: 2D spatial blocks + 1D temporal blocks
  • Latent: [batch, latentChannels, frames, height/8, width/8]
  • Decoder: 2D spatial blocks + 1D temporal blocks
  • Output: [batch, channels, frames, height, width] reconstructed video

Example

var vae = new TemporalVAE<float>(inputChannels: 3, latentChannels: 4);
var video = Tensor<float>.CreateRandom(new[] { 1, 3, 14, 320, 512 });
var latent = vae.Encode(video);

Constructors

Constructor Summary
TemporalVAE(int inputChannels, int latentChannels, int baseChannels, int[] channelMultipliers, int numTemporalLayers, int temporalKernelSize, bool causalMode, double? latentScaleFactor, ILossFunction<T> lossFunction, int? seed) Initializes a new instance of the TemporalVAE class.

Properties

Property Summary
bool IsCausal { get; } Gets whether this VAE uses causal convolutions.
bool SupportsSlicing { get; }
bool SupportsTiling { get; }
double LatentScaleFactor { get; }
int DownsampleFactor { get; }
int InputChannels { get; }
int LatentChannels { get; }
int TemporalKernelSize { get; } Gets the temporal kernel size.

Methods

Method Summary
Tensor<T> Decode(Tensor<T> latent)
Tensor<T> DecodeVideoFromDiffusion(Tensor<T> latent) Decodes a diffusion video latent back to video space.
Tensor<T> Encode(Tensor<T> video, bool sampleMode)
Tensor<T> EncodeVideoForDiffusion(Tensor<T> video, bool sampleMode) Encodes a video and applies latent scaling for use in diffusion.
ValueTuple<Tensor<T>, Tensor<T>> EncodeWithDistribution(Tensor<T> video)