LatteModel<T>

Models & Types · AiDotNet.Diffusion.Video

Latte model for Latent Diffusion Transformer video generation with factorized spatial-temporal attention.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → VideoDiffusionModelBase<T> → LatteModel<T>

For Beginners

Latte applies DiT (Diffusion Transformer) concepts to video generation.

How Latte works:

  1. Text prompt is encoded by T5-XXL into 4096-dim embeddings
  2. Each video frame is encoded by the SD VAE into 4 latent channels
  3. Latent frames are patchified and processed by the DiT with factorized attention
  4. Spatial attention handles per-frame content, temporal attention handles motion
  5. The VAE decodes each latent frame back to pixel space

Key characteristics:

  • Explores 4 attention decomposition strategies for efficiency
  • “Decomposed” variant (spatial then temporal) achieves best quality/speed
  • Uses per-frame VAE (standard SD VAE, not temporal)
  • 16 frames at 8 FPS by default (~2 seconds)
  • ~700M parameter DiT backbone

When to use Latte:

  • Research on efficient video DiT architectures
  • Short clip generation from text prompts
  • Exploring spatial-temporal attention decomposition
  • Lightweight alternative to full 3D attention models

Limitations:

  • Per-frame VAE may cause minor temporal artifacts
  • Shorter duration than temporal-VAE-based models
  • Lower resolution than SDXL-based video models
  • Research model, not production-grade quality

How It Works

Latte applies Diffusion Transformer (DiT) architecture to video generation by exploring factorized spatial-temporal attention patterns within transformer blocks. The model decomposes full 3D attention into efficient spatial and temporal components.

Architecture components:

  • DiT backbone with 28 transformer layers and 1152 hidden dimension
  • 4 attention variants: joint, spatial-first, temporal-first, decomposed
  • 16 attention heads with efficient O(n) factorized attention
  • T5-XXL text encoder for 4096-dim context embeddings
  • Standard SD VAE for per-frame spatial compression (4 latent channels)
  • Patch size 2 for spatiotemporal tokenization

Technical specifications:

  • Architecture: DiT with factorized spatial-temporal attention
  • Hidden dimension: 1152
  • Transformer layers: 28
  • Attention heads: 16
  • Patch size: 2
  • Latent channels: 4 (standard SD VAE)
  • Context dimension: 4096 (T5-XXL)
  • Default: 16 frames at 8 FPS (~2 seconds)
  • Noise schedule: Linear beta [0.0001, 0.02], 1000 timesteps
  • Scheduler: DDIM
  • Parameters: ~700M (DiT backbone)

Reference: Ma et al., “Latte: Latent Diffusion Transformer for Video Generation”, 2024

Example

// Create with industry-standard defaults
var latte = new LatteModel<float>();
            
// Generate video from text
var video = latte.GenerateFromText(
    prompt: "A cat sitting on a windowsill watching birds",
    width: 512,
    height: 512,
    numFrames: 16,
    fps: 8,
    numInferenceSteps: 50,
    guidanceScale: 7.5);

Constructors

Constructor Summary
LatteModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, DiTNoisePredictor<T> dit, StandardVAE<T> vae, IConditioningModule<T> conditioner, int defaultNumFrames, int defaultFPS, int? seed) Initializes a new instance of LatteModel with full customization support.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> VAE { get; }
bool SupportsImageToVideo { get; }
bool SupportsTextToVideo { get; }
bool SupportsVideoToVideo { get; }
int LatentChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()