VeoModel<T>

Models & Types · AiDotNet.Diffusion.Video

Veo model for Google’s high-fidelity cascaded video generation with temporal super-resolution.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → VideoDiffusionModelBase<T> → VeoModel<T>

For Beginners

Veo is Google’s top-tier video generation model.

How Veo works:

  1. Text is encoded by dual T5-XXL + CLIP encoders into 4096-dim embeddings
  2. Base model generates low-resolution video in compressed latent space
  3. Spatial super-resolution stage upscales each frame
  4. Temporal super-resolution stage adds interpolated frames
  5. The causal VAE decodes the final high-resolution video

Key characteristics:

  • Cascaded architecture: base → spatial SR → temporal SR
  • 1080p output with 60+ second duration capability
  • Dual text encoding (T5-XXL + CLIP) for rich conditioning
  • Veo 2 variant with improved quality and consistency
  • 150 frames at 24 FPS by default (~6.25 seconds)

When to use Veo:

  • Highest-quality video generation
  • Long-duration video content
  • 1080p high-resolution output
  • Text-to-video, image-to-video, and video-to-video tasks

Limitations:

  • Proprietary model (API-only access through Google)
  • Very high compute requirements for cascaded generation
  • Slower generation due to multi-stage pipeline
  • Limited public information on exact architecture details

How It Works

Veo by Google DeepMind uses cascaded diffusion with temporal super-resolution for high-resolution, long-duration video generation. The base model generates at lower resolution, then spatial and temporal super-resolution stages produce the final output.

Architecture components:

  • Cascaded DiT with 40 transformer layers and 2560 hidden dimension
  • 20 attention heads with full spatiotemporal attention
  • T5-XXL + CLIP dual text encoding for 4096-dim context
  • 3D causal VAE with 16 latent channels and 3 temporal layers
  • Cascaded pipeline: base → spatial SR → temporal SR
  • Flow matching training objective

Technical specifications:

  • Architecture: Cascaded DiT (base + spatial SR + temporal SR)
  • Hidden dimension: 2560
  • Transformer layers: 40
  • Attention heads: 20
  • Patch size: 2
  • Latent channels: 16 (3D causal VAE)
  • Context dimension: 4096 (T5-XXL + CLIP dual encoder)
  • Output resolution: Up to 1080p
  • Default: 150 frames at 24 FPS (~6.25 seconds)
  • Veo 2: Enhanced quality variant (200 frames default)
  • Training objective: Flow matching

Reference: Google DeepMind, “Veo: High-Fidelity Video Generation”, 2024

Example

var referenceFrame = Tensor<float>.CreateRandom(1, 3, 32, 32);
// Create Veo with defaults
var veo = new VeoModel<float>();
            
// Create Veo 2 variant
var veo2 = VeoModel<float>.CreateVeo2();
            
// Generate video from text
var video = veo.GenerateFromText(
    prompt: "An aerial view of a coral reef teeming with tropical fish",
    width: 1920,
    height: 1080,
    numFrames: 150,
    fps: 24,
    numInferenceSteps: 50,
    guidanceScale: 7.5);
            
// Generate video from an image
var animated = veo.GenerateFromImage(
    inputImage: referenceFrame,
    numFrames: 150,
    numInferenceSteps: 50);

Constructors

Constructor Summary
VeoModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, DiTNoisePredictor<T> dit, TemporalVAE<T> temporalVAE, IConditioningModule<T> conditioner, bool isVeo2, int defaultNumFrames, int defaultFPS, int? seed) Initializes a new instance of VeoModel with full customization support.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> TemporalVAE { get; }
IVAEModel<T> VAE { get; }
bool IsVeo2 { get; } Gets whether this is a Veo 2 variant with enhanced quality.
bool SupportsImageToVideo { get; }
bool SupportsTextToVideo { get; }
bool SupportsVideoToVideo { get; }
int LatentChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
VeoModel<T> CreateVeo2(IConditioningModule<T> conditioner) Creates a Veo 2 variant with enhanced quality and longer duration.