CogVideoModel<T>
Models & Types · AiDotNet.Diffusion.Video
CogVideo/CogVideoX model for text-to-video and image-to-video generation.
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → VideoDiffusionModelBase<T> → CogVideoModel<T>
For Beginners
CogVideoX generates videos from text descriptions:
How CogVideoX works:
- Text is encoded by a T5 text encoder
- A 3D causal VAE encodes/decodes video in compressed latent space
- A diffusion transformer (DiT) denoises the video latents
- The 3D VAE decodes latents back to video frames
Key characteristics:
- 3D causal VAE for temporal compression (4x temporal, 8x spatial)
- Expert transformer blocks with adaptive layer norm
- T5 text encoder for text understanding
- CogVideoX-2B: 2B parameters, 480p output
- CogVideoX-5B: 5B parameters, 720p output
- 49 frames at 8 FPS (~6 seconds)
Advantages:
- Open-source with permissive license
- Strong temporal coherence
- Good prompt adherence
- Efficient 3D VAE compression
Limitations:
- Generation is slow (~minutes per video)
- Requires significant VRAM
- Limited to short clips
How It Works
CogVideoX is a large-scale text-to-video generation model developed by Zhipu AI / THUDM. It uses a 3D causal VAE and a transformer-based architecture for generating coherent video.
Technical specifications:
- Architecture: 3D Causal VAE + Diffusion Transformer
- CogVideoX-2B: 2B parameters, 480×720
- CogVideoX-5B: 5B parameters, 480×720
- 3D Causal VAE: 4x temporal + 8x spatial compression, 16 latent channels
- Text encoder: T5-XXL (4096-dim)
- Frames: 49 at 8 FPS
- Noise schedule: Scaled linear, 1000 training timesteps
Reference: Hong et al., “CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers”, ICLR 2023
Example
var cogvideo = new CogVideoModel<float>();
var video = cogvideo.GenerateFromText(
prompt: "A cat playing piano",
numFrames: 49,
fps: 8,
numInferenceSteps: 50,
guidanceScale: 6.0,
seed: 42);
Constructors
| Constructor | Summary |
|---|---|
CogVideoModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, VideoUNetPredictor<T> videoUnet, TemporalVAE<T> temporalVae, IConditioningModule<T> conditioner, string variant, int defaultNumFrames, int defaultFPS, int? seed) |
Initializes a new instance of CogVideoModel with full customization support. |
Properties
| Property | Summary |
|---|---|
IConditioningModule<T> Conditioner { get; } |
|
INoisePredictor<T> NoisePredictor { get; } |
|
IVAEModel<T> TemporalVAE { get; } |
|
IVAEModel<T> VAE { get; } |
|
bool SupportsImageToVideo { get; } |
|
bool SupportsTextToVideo { get; } |
|
bool SupportsVideoToVideo { get; } |
|
int LatentChannels { get; } |
|
string Variant { get; } |
Gets the model variant (“2B” or “5B”). |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
Fields
| Field | Summary |
|---|---|
int DefaultHeight |
|
int DefaultWidth |