LatteModel<T>
Models & Types · AiDotNet.Diffusion.Video
Latte model for Latent Diffusion Transformer video generation with factorized spatial-temporal attention.
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → VideoDiffusionModelBase<T> → LatteModel<T>
For Beginners
Latte applies DiT (Diffusion Transformer) concepts to video generation.
How Latte works:
- Text prompt is encoded by T5-XXL into 4096-dim embeddings
- Each video frame is encoded by the SD VAE into 4 latent channels
- Latent frames are patchified and processed by the DiT with factorized attention
- Spatial attention handles per-frame content, temporal attention handles motion
- The VAE decodes each latent frame back to pixel space
Key characteristics:
- Explores 4 attention decomposition strategies for efficiency
- “Decomposed” variant (spatial then temporal) achieves best quality/speed
- Uses per-frame VAE (standard SD VAE, not temporal)
- 16 frames at 8 FPS by default (~2 seconds)
- ~700M parameter DiT backbone
When to use Latte:
- Research on efficient video DiT architectures
- Short clip generation from text prompts
- Exploring spatial-temporal attention decomposition
- Lightweight alternative to full 3D attention models
Limitations:
- Per-frame VAE may cause minor temporal artifacts
- Shorter duration than temporal-VAE-based models
- Lower resolution than SDXL-based video models
- Research model, not production-grade quality
How It Works
Latte applies Diffusion Transformer (DiT) architecture to video generation by exploring factorized spatial-temporal attention patterns within transformer blocks. The model decomposes full 3D attention into efficient spatial and temporal components.
Architecture components:
- DiT backbone with 28 transformer layers and 1152 hidden dimension
- 4 attention variants: joint, spatial-first, temporal-first, decomposed
- 16 attention heads with efficient O(n) factorized attention
- T5-XXL text encoder for 4096-dim context embeddings
- Standard SD VAE for per-frame spatial compression (4 latent channels)
- Patch size 2 for spatiotemporal tokenization
Technical specifications:
- Architecture: DiT with factorized spatial-temporal attention
- Hidden dimension: 1152
- Transformer layers: 28
- Attention heads: 16
- Patch size: 2
- Latent channels: 4 (standard SD VAE)
- Context dimension: 4096 (T5-XXL)
- Default: 16 frames at 8 FPS (~2 seconds)
- Noise schedule: Linear beta [0.0001, 0.02], 1000 timesteps
- Scheduler: DDIM
- Parameters: ~700M (DiT backbone)
Reference: Ma et al., “Latte: Latent Diffusion Transformer for Video Generation”, 2024
Example
// Create with industry-standard defaults
var latte = new LatteModel<float>();
// Generate video from text
var video = latte.GenerateFromText(
prompt: "A cat sitting on a windowsill watching birds",
width: 512,
height: 512,
numFrames: 16,
fps: 8,
numInferenceSteps: 50,
guidanceScale: 7.5);
Constructors
| Constructor | Summary |
|---|---|
LatteModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, DiTNoisePredictor<T> dit, StandardVAE<T> vae, IConditioningModule<T> conditioner, int defaultNumFrames, int defaultFPS, int? seed) |
Initializes a new instance of LatteModel with full customization support. |
Properties
| Property | Summary |
|---|---|
IConditioningModule<T> Conditioner { get; } |
|
INoisePredictor<T> NoisePredictor { get; } |
|
IVAEModel<T> VAE { get; } |
|
bool SupportsImageToVideo { get; } |
|
bool SupportsTextToVideo { get; } |
|
bool SupportsVideoToVideo { get; } |
|
int LatentChannels { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |