CogVideoModel<T>

Models & Types · AiDotNet.Diffusion.Video

CogVideo/CogVideoX model for text-to-video and image-to-video generation.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → VideoDiffusionModelBase<T> → CogVideoModel<T>

For Beginners

CogVideoX generates videos from text descriptions:

How CogVideoX works:

  1. Text is encoded by a T5 text encoder
  2. A 3D causal VAE encodes/decodes video in compressed latent space
  3. A diffusion transformer (DiT) denoises the video latents
  4. The 3D VAE decodes latents back to video frames

Key characteristics:

  • 3D causal VAE for temporal compression (4x temporal, 8x spatial)
  • Expert transformer blocks with adaptive layer norm
  • T5 text encoder for text understanding
  • CogVideoX-2B: 2B parameters, 480p output
  • CogVideoX-5B: 5B parameters, 720p output
  • 49 frames at 8 FPS (~6 seconds)

Advantages:

  • Open-source with permissive license
  • Strong temporal coherence
  • Good prompt adherence
  • Efficient 3D VAE compression

Limitations:

  • Generation is slow (~minutes per video)
  • Requires significant VRAM
  • Limited to short clips

How It Works

CogVideoX is a large-scale text-to-video generation model developed by Zhipu AI / THUDM. It uses a 3D causal VAE and a transformer-based architecture for generating coherent video.

Technical specifications:

  • Architecture: 3D Causal VAE + Diffusion Transformer
  • CogVideoX-2B: 2B parameters, 480×720
  • CogVideoX-5B: 5B parameters, 480×720
  • 3D Causal VAE: 4x temporal + 8x spatial compression, 16 latent channels
  • Text encoder: T5-XXL (4096-dim)
  • Frames: 49 at 8 FPS
  • Noise schedule: Scaled linear, 1000 training timesteps

Reference: Hong et al., “CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers”, ICLR 2023

Example

var cogvideo = new CogVideoModel<float>();
var video = cogvideo.GenerateFromText(
    prompt: "A cat playing piano",
    numFrames: 49,
    fps: 8,
    numInferenceSteps: 50,
    guidanceScale: 6.0,
    seed: 42);

Constructors

Constructor Summary
CogVideoModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, VideoUNetPredictor<T> videoUnet, TemporalVAE<T> temporalVae, IConditioningModule<T> conditioner, string variant, int defaultNumFrames, int defaultFPS, int? seed) Initializes a new instance of CogVideoModel with full customization support.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> TemporalVAE { get; }
IVAEModel<T> VAE { get; }
bool SupportsImageToVideo { get; }
bool SupportsTextToVideo { get; }
bool SupportsVideoToVideo { get; }
int LatentChannels { get; }
string Variant { get; } Gets the model variant (“2B” or “5B”).

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()

Fields

Field Summary
int DefaultHeight
int DefaultWidth