KlingModel<T>

Models & Types · AiDotNet.Diffusion.Video

Kling model — 3D spatiotemporal attention video generation by Kuaishou.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → VideoDiffusionModelBase<T> → KlingModel<T>

For Beginners

Kling is a video generation model from Kuaishou that creates high-quality videos from text or images.

How Kling works:

  1. Text prompt is encoded into a 4096-dimensional embedding
  2. A 3D DiT transformer generates video in compressed latent space
  3. Full 3D attention (not factorized) captures spatial AND temporal relationships
  4. A temporal VAE decompresses the latent video into pixel-space frames

Advantages:

  • Very high video quality at 1080p resolution
  • Long video support (up to 2 minutes)
  • Strong physics and motion understanding
  • Both text-to-video and image-to-video generation

Limitations:

  • Very large model requiring significant GPU memory
  • Proprietary architecture with limited public details
  • Slower generation than smaller video models

How It Works

Kling uses a 3D VAE with full spatiotemporal attention for high-quality video generation with strong motion consistency and physics understanding. It supports up to 2 minutes of video at 1080p resolution.

Architecture components:

  • DiT backbone with 3D full attention (36 layers, 2048 hidden dim)
  • Temporal VAE for spatiotemporal compression (causal mode)
  • Large-scale text encoder (4096-dim context)
  • 16-channel latent space for high-fidelity reconstruction

Technical specifications:

  • Architecture: DiT with full 3D spatiotemporal attention
  • Parameters: ~5B+ (estimated)
  • Backbone: 36 transformer layers, 2048 hidden dim, 16 attention heads
  • Latent space: 16 channels with temporal VAE
  • Text encoder: 4096-dimensional context embedding
  • Max resolution: 1080p (1920x1080)
  • Max duration: 2 minutes at 30 FPS
  • Noise schedule: Flow matching

Reference: Kuaishou, “Kling: A Text-to-Video Generation Model”, 2024

Example

var referenceImage = Tensor<float>.CreateRandom(1, 3, 32, 32);
// Create with default settings
var kling = new KlingModel<float>();
            
// Generate video from text
var frames = kling.GenerateFromText(
    prompt: "A drone flying over a mountain landscape at sunset",
    numFrames: 60,
    numInferenceSteps: 50,
    guidanceScale: 7.0);
            
// Generate video from an image
var animatedFrames = kling.GenerateFromImage(
    inputImage: referenceImage,
    numFrames: 30);

Constructors

Constructor Summary
KlingModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, DiTNoisePredictor<T> dit, TemporalVAE<T> temporalVAE, IConditioningModule<T> conditioner, int defaultNumFrames, int defaultFPS, int? seed) Initializes a new instance of KlingModel with full customization support.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> TemporalVAE { get; }
IVAEModel<T> VAE { get; }
bool SupportsImageToVideo { get; }
bool SupportsTextToVideo { get; }
bool SupportsVideoToVideo { get; }
int LatentChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()