TemporalVAE<T>
Models & Types · AiDotNet.Diffusion.VAE
Temporal-aware Variational Autoencoder for video diffusion models.
Inheritance: VAEModelBase<T> → TemporalVAE<T>
For Beginners
While a standard VAE processes each frame independently, TemporalVAE considers relationships between consecutive frames:
Standard VAE approach (per-frame):
- Frame 1 -> Latent 1 (no knowledge of other frames)
- Frame 2 -> Latent 2 (no knowledge of other frames)
- Result: Possible flickering/inconsistency between frames
TemporalVAE approach:
- Frames 1,2,3,… -> Encode with temporal awareness
- Latent knows about neighboring frames
- Result: Smoother, more consistent video
Key features:
- 3D convolutions that span across time dimension
- Temporal attention for long-range frame relationships
- Optional causal mode for streaming/autoregressive generation
Used in: Stable Video Diffusion, Video LDM, and similar models.
How It Works
The TemporalVAE extends the standard VAE to handle video data by incorporating temporal awareness into the encoding and decoding process. This helps maintain temporal consistency across frames when used in video diffusion models.
Architecture details:
- Input: [batch, channels, frames, height, width] video tensor
- Encoder: 2D spatial blocks + 1D temporal blocks
- Latent: [batch, latentChannels, frames, height/8, width/8]
- Decoder: 2D spatial blocks + 1D temporal blocks
- Output: [batch, channels, frames, height, width] reconstructed video
Example
var vae = new TemporalVAE<float>(inputChannels: 3, latentChannels: 4);
var video = Tensor<float>.CreateRandom(new[] { 1, 3, 14, 320, 512 });
var latent = vae.Encode(video);
Constructors
| Constructor | Summary |
|---|---|
TemporalVAE(int inputChannels, int latentChannels, int baseChannels, int[] channelMultipliers, int numTemporalLayers, int temporalKernelSize, bool causalMode, double? latentScaleFactor, ILossFunction<T> lossFunction, int? seed) |
Initializes a new instance of the TemporalVAE class. |
Properties
| Property | Summary |
|---|---|
bool IsCausal { get; } |
Gets whether this VAE uses causal convolutions. |
bool SupportsSlicing { get; } |
|
bool SupportsTiling { get; } |
|
double LatentScaleFactor { get; } |
|
int DownsampleFactor { get; } |
|
int InputChannels { get; } |
|
int LatentChannels { get; } |
|
int TemporalKernelSize { get; } |
Gets the temporal kernel size. |
Methods
| Method | Summary |
|---|---|
Tensor<T> Decode(Tensor<T> latent) |
|
Tensor<T> DecodeVideoFromDiffusion(Tensor<T> latent) |
Decodes a diffusion video latent back to video space. |
Tensor<T> Encode(Tensor<T> video, bool sampleMode) |
|
Tensor<T> EncodeVideoForDiffusion(Tensor<T> video, bool sampleMode) |
Encodes a video and applies latent scaling for use in diffusion. |
ValueTuple<Tensor<T>, Tensor<T>> EncodeWithDistribution(Tensor<T> video) |