CogVideoX15Model<T>

Models & Types · AiDotNet.Diffusion.Video

CogVideoX 1.5 model with 10-second any-resolution video generation.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → VideoDiffusionModelBase<T> → CogVideoX15Model<T>

For Beginners

CogVideoX 1.5 creates 10-second videos at any resolution from text prompts. It compresses video efficiently using a 3D causal VAE (reducing memory 128x) and uses specialized expert transformer blocks that understand both spatial layout and temporal motion.

How It Works

References:

  • Paper: “CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer” (Zhipu AI/THUDM, 2024)

CogVideoX 1.5 extends the original CogVideoX to 10-second generation with any-resolution support. Uses 3D causal VAE with 4x temporal and 8x spatial compression, expert transformer blocks with adaptive layer normalization, and T5 text encoding. The 5B parameter model supports both text-to-video and image-to-video generation at up to 720p.

Technical specifications:

  • Architecture: 3D Causal VAE + Expert Transformer + T5
  • Latent channels: 16
  • Default: 80 frames at 8 FPS
  • Supports I2V: Yes | T2V: Yes | V2V: No

Example

var options = new DiffusionModelOptions<float> { LatentChannels = 16, DefaultInferenceSteps = 50 };
var noise = Tensor<float>.CreateRandom(new[] { 1, 16, 80, 60, 90 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new CogVideoX15Model<float>(options: options))
    .Build(trainX, trainY);
var video = result.Predict(noise);

Constructors

Constructor Summary
CogVideoX15Model(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, DiTNoisePredictor<T> predictor, TemporalVAE<T> temporalVAE, IConditioningModule<T> conditioner, int defaultNumFrames, int defaultFPS, int? seed) Initializes a new instance of CogVideoX15Model with full customization support.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> TemporalVAE { get; }
IVAEModel<T> VAE { get; }
bool SupportsImageToVideo { get; }
bool SupportsTextToVideo { get; }
bool SupportsVideoToVideo { get; }
int LatentChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()