ImprovedVideoVAE<T>

Models & Types · AiDotNet.Diffusion.VAE

Improved Video VAE with temporal-aware compression and motion consistency.

Inheritance: VAEModelBase<T> → ImprovedVideoVAE<T>

Implements: IGeneratedParameterRegistrar<T>

For Beginners

A video is like a stack of images. This VAE compresses both spatially (each frame gets smaller) and temporally (every 4 frames become 1 latent frame). It understands motion — if a ball is moving right, the latent representation captures that smoothly rather than treating each frame independently.

How It Works

Extends standard image VAE with temporal convolutions and motion-aware encoding. Uses causal temporal convolutions (only looking at past frames) for streaming-compatible encoding, and temporal attention for capturing long-range motion patterns. Achieves 4x temporal compression on top of 8x spatial compression.

Reference: Improved upon CogVideoX VAE architecture with motion-aware encoding, 2024-2025

Example

var vae = new ImprovedVideoVAE<float>(inputChannels: 3, latentChannels: 4);
var video = Tensor<float>.CreateRandom(new[] { 1, 3, 16, 256, 256 });
var latent = vae.Encode(video);

Constructors

Constructor Summary
ImprovedVideoVAE(int inputChannels, int latentChannels, int baseChannels, int temporalDownsample, ILossFunction<T> lossFunction, int? seed) Initializes a new Improved Video VAE.

Properties

Property Summary
bool SupportsSlicing { get; }
bool SupportsTiling { get; }
double LatentScaleFactor { get; }
int DownsampleFactor { get; }
int InputChannels { get; }
int LatentChannels { get; }
int TemporalDownsampleFactor { get; } Gets the temporal downsampling factor.

Methods

Method Summary
List<Tensor<T>> EncodeVideo(List<Tensor<T>> frames) Encodes a video (sequence of frames) with temporal compression.
Tensor<T> Decode(Tensor<T> latent)
Tensor<T> Encode(Tensor<T> image, bool sampleMode)
ValueTuple<Tensor<T>, Tensor<T>> EncodeWithDistribution(Tensor<T> image)