Causal3DVAE<T>
Models & Types · AiDotNet.Diffusion.VAE
CogVideoX’s causal 3D VAE: video compressed 8x in height and width and 4x in time, every convolution causal in time.
Inheritance: VAEModelBase<T> → Causal3DVAE<T>
Implements: IVideoVAEModel<T>
For Beginners
The Causal 3D VAE compresses video into a much smaller latent space while preserving temporal information. Causal means every convolution reads only the current and earlier frames, so the first frame keeps its own latent frame and a still image is simply a one-frame clip. (The group normalizations take their statistics over the whole clip, as the reference does, so a later frame still shifts earlier latents slightly.)
How It Works
References:
- Paper: “CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer” (Yang et al., 2024), section 2.1
- Implementation: diffusers
AutoencoderKLCogVideoX, which this reproduces (see
tools/reference-data/cogvideox_vae_reference.py); LoadCogVideoXWeights loads its state dict.
Architecture (the paper’s figure 3, diffusers’ layout):
- Encoder: a causal 3x3x3 conv, one stage of residual blocks per width in
blockOutChannels, each stage but
the last halving height and width (a stride-2 conv) and the first log2(temporalCompression) also averaging frame pairs; two residual blocks; GroupNorm, SiLU, and a causal conv to the posterior’s mean and log-variance.
- Decoder: the mirror, with one more residual block per stage, nearest-neighbour upsampling, and every
normalization a spatial norm modulated by the latent (MoVQ’s SpatialNorm).
- Causal padding repeats the first frame in front, so a clip of 1 + 4k frames encodes to 1 + k latent frames.
An image, [batch, channels, height, width], is encoded as a one-frame clip and its latent is returned without the frame axis. diffusers encodes long clips in 8-frame batches carrying a convolution cache to bound memory; this model encodes the clip whole, which is the network both describe.
Example
var vae = new Causal3DVAE<float>();
var video = Tensor<float>.CreateRandom(new[] { 1, 3, 9, 64, 64 });
var latent = vae.Encode(video, sampleMode: false); // [1, 16, 3, 8, 8]
Constructors
| Constructor | Summary |
|---|---|
Causal3DVAE(int inputChannels, int latentChannels, int[] blockOutChannels, int layersPerBlock, int normGroups, int temporalCompression, double latentScaleFactor, ILossFunction<T> lossFunction, int? seed) |
Initializes a new Causal 3D VAE. |
Properties
| Property | Summary |
|---|---|
double LatentScaleFactor { get; } |
|
int DownsampleFactor { get; } |
|
int InputChannels { get; } |
|
int LatentChannels { get; } |
|
int TemporalCompression { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyCollection<string> LoadCogVideoXWeights(Func<string, int[], double[]> read) |
Loads a diffusers AutoencoderKLCogVideoX state dict straight from source into the weights. |
Tensor<T> Decode(Tensor<T> latent) |
|
Tensor<T> Encode(Tensor<T> input, bool sampleMode) |
|
ValueTuple<Tensor<T>, Tensor<T>> EncodeWithDistribution(Tensor<T> input) |
|
int GetLatentFrameCount(int frames) |