HunyuanVideoModel<T>

Models & Types · AiDotNet.Diffusion.Video

HunyuanVideo model for dual-stream DiT video generation with unified image-video capability.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → VideoDiffusionModelBase<T> → HunyuanVideoModel<T>

For Beginners

HunyuanVideo is Tencent’s open-source video generation model.

How HunyuanVideo works:

  1. Text prompt is encoded by an MLLM-based encoder into 4096-dim embeddings
  2. Video is compressed by the 3D causal VAE into a 16-channel latent (4x4x4 compression)
  3. The DS-DiT processes text and video in dual streams, then merges for joint attention
  4. Flow matching denoises the video latent over the scheduled timesteps
  5. The causal VAE decodes the latent back to 720p video

Key characteristics:

  • 13B parameters total (one of the largest open-source video models)
  • 720p resolution, 5+ second duration, 129 frames at 24 FPS
  • Open-source weights available
  • Unified image-video generation (both supported)
  • 3D causal VAE enables temporal consistency

When to use HunyuanVideo:

  • High-quality open-source video generation
  • Text-to-video with strong prompt adherence
  • Image-to-video animation
  • Research and experimentation with large video models

Limitations:

  • Very large model (13B parameters, requires significant GPU memory)
  • Slower generation than lightweight models
  • Causal VAE may have slight quality loss at clip boundaries

How It Works

HunyuanVideo by Tencent uses a “Dual-stream to Single-stream” Diffusion Transformer (DS-DiT) architecture with a 3D causal VAE for high-resolution video generation. The dual-stream design processes text and video tokens separately in early layers, then merges them for joint attention in later layers.

Architecture components:

  • DS-DiT (Dual-stream to Single-stream DiT) with 40 transformer layers
  • 3072 hidden dimension with 24 attention heads
  • 3D causal VAE with 4x4x4 spatiotemporal compression (16 latent channels)
  • MLLM-based text encoder for rich semantic understanding (4096-dim context)
  • Flow matching training objective for stable convergence

Technical specifications:

  • Architecture: DS-DiT (Dual-stream to Single-stream Diffusion Transformer)
  • Hidden dimension: 3072
  • Transformer layers: 40 (dual-stream early, single-stream late)
  • Attention heads: 24
  • Patch size: 2x2 spatiotemporal patches
  • Latent channels: 16 (compressed by 3D causal VAE)
  • Context dimension: 4096 (MLLM text encoder)
  • VAE compression: 4x4x4 spatiotemporal
  • Default: 129 frames at 24 FPS (~5.4 seconds)
  • Training objective: Flow matching
  • Total parameters: ~13B

Reference: Kong et al., “HunyuanVideo: A Systematic Framework For Large Video Generative Models”, 2024

Example

var referenceFrame = Tensor<float>.CreateRandom(1, 3, 32, 32);
// Create with industry-standard defaults
var hunyuan = new HunyuanVideoModel<float>();
            
// Generate video from text (requires conditioner)
var video = hunyuan.GenerateFromText(
    prompt: "A golden retriever running through a sunlit meadow",
    width: 1280,
    height: 720,
    numFrames: 129,
    fps: 24,
    numInferenceSteps: 50,
    guidanceScale: 6.0);
            
// Generate video from an image
var animated = hunyuan.GenerateFromImage(
    inputImage: referenceFrame,
    numFrames: 129,
    numInferenceSteps: 50);

Constructors

Constructor Summary
HunyuanVideoModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, DiTNoisePredictor<T> dit, TemporalVAE<T> temporalVAE, IConditioningModule<T> conditioner, int defaultNumFrames, int defaultFPS, int? seed) Initializes a new instance of HunyuanVideoModel with full customization support.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> TemporalVAE { get; }
IVAEModel<T> VAE { get; }
bool SupportsImageToVideo { get; }
bool SupportsTextToVideo { get; }
bool SupportsVideoToVideo { get; }
int LatentChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()