LTXVideoModel<T>

Models & Types · AiDotNet.Diffusion.Video

LTX-Video model for lightweight real-time video generation with extreme latent compression.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → VideoDiffusionModelBase<T> → LTXVideoModel<T>

For Beginners

LTX-Video generates videos faster than real-time by compressing heavily.

How LTX-Video works:

  1. Text prompt is encoded by T5-XXL into 4096-dim embeddings
  2. Video is compressed by the 3D causal VAE into 128-channel latent (192x compression)
  3. The lightweight DiT denoises the latent using flow matching
  4. The causal VAE decodes the latent back to 720p video

Key characteristics:

  • ~2B parameters (lightweight for a video model)
  • 192x spatiotemporal compression (highest in class)
  • 128 latent channels for rich representations despite compression
  • 720p, 5 seconds at 24 FPS, faster-than-real-time generation
  • Open-source with Lightricks weights

When to use LTX-Video:

  • Real-time or interactive video generation
  • Applications requiring low latency
  • Edge deployment with limited compute
  • Rapid prototyping and iteration

Limitations:

  • High compression may lose fine spatial details
  • Quality may not match larger models (Sora, HunyuanVideo)
  • Best for medium-resolution content
  • Trade-off between speed and fidelity

How It Works

LTX-Video by Lightricks is designed for efficient video generation, using a lightweight DiT transformer operating in a highly compressed latent space via a 3D causal VAE. The extreme 192x compression ratio enables faster-than-real-time generation.

Architecture components:

  • Lightweight DiT with 28 transformer layers and 1536 hidden dimension
  • 16 attention heads for multi-head self-attention
  • 3D causal VAE with 128 latent channels and 192x compression ratio
  • T5-XXL text encoder for 4096-dim context embeddings
  • Flow matching training objective for stable convergence
  • Patch size 1 (operates on individual latent tokens)

Technical specifications:

  • Architecture: Lightweight DiT with 3D causal VAE
  • Hidden dimension: 1536
  • Transformer layers: 28
  • Attention heads: 16
  • Patch size: 1 (direct latent token processing)
  • Latent channels: 128 (extreme compression)
  • Context dimension: 4096 (T5-XXL)
  • VAE compression: 192x spatiotemporal
  • Default: 121 frames at 24 FPS (~5 seconds)
  • Training objective: Flow matching
  • Total parameters: ~2B

Reference: Lightricks, “LTX-Video: Realtime Video Latent Diffusion”, 2024

Example

var referenceFrame = Tensor<float>.CreateRandom(1, 3, 32, 32);
// Create with industry-standard defaults
var ltxVideo = new LTXVideoModel<float>();
            
// Generate video from text (faster than real-time)
var video = ltxVideo.GenerateFromText(
    prompt: "A time-lapse of clouds moving across a mountain range",
    width: 1280,
    height: 720,
    numFrames: 121,
    fps: 24,
    numInferenceSteps: 30,
    guidanceScale: 7.0);
            
// Generate video from an image
var animated = ltxVideo.GenerateFromImage(
    inputImage: referenceFrame,
    numFrames: 121,
    numInferenceSteps: 30);

Constructors

Constructor Summary
LTXVideoModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, DiTNoisePredictor<T> dit, TemporalVAE<T> temporalVAE, IConditioningModule<T> conditioner, int defaultNumFrames, int defaultFPS, int? seed) Initializes a new instance of LTXVideoModel with full customization support.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> TemporalVAE { get; }
IVAEModel<T> VAE { get; }
bool SupportsImageToVideo { get; }
bool SupportsTextToVideo { get; }
bool SupportsVideoToVideo { get; }
int LatentChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()