OpenSoraModel<T>

Models & Types · AiDotNet.Diffusion.Video

Open-Sora model for open-source Sora-like video generation with STDiT architecture.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → VideoDiffusionModelBase<T> → OpenSoraModel<T>

For Beginners

Open-Sora is an open-source alternative to OpenAI’s Sora.

How Open-Sora works:

  1. Text prompt is encoded by T5-XXL into 4096-dim embeddings
  2. Video is compressed by the 3D causal VAE into 4-channel latent space
  3. The STDiT applies spatial attention first, then temporal attention per layer
  4. Rectified flow denoises the video latent over scheduled timesteps
  5. The causal VAE decodes the latent back to video frames

Key characteristics:

  • Open-source community project replicating Sora capabilities
  • STDiT factorizes spatial and temporal attention for efficiency
  • Rectified flow training (straighter trajectories than DDPM)
  • Multi-resolution training enables variable aspect ratio/duration
  • 51 frames at 24 FPS by default (~2.1 seconds)

When to use Open-Sora:

  • Open-source video generation research
  • Variable-resolution video generation
  • Text-to-video, image-to-video, and video-to-video tasks
  • Community-driven experimentation and fine-tuning

Limitations:

  • Quality below commercial models (Sora, Veo)
  • Research-stage model with ongoing improvements
  • Limited duration compared to larger models
  • Requires significant compute for training

How It Works

Open-Sora is an open-source reproduction of Sora-like capabilities using the Spatial-Temporal DiT (STDiT) architecture. It features efficient spatial-temporal attention factorization, rectified flow training, and multi-resolution masking strategy.

Architecture components:

  • STDiT (Spatial-Temporal Diffusion Transformer) with 28 layers
  • 1152 hidden dimension with 16 attention heads
  • 3D causal VAE with 4 latent channels for spatiotemporal compression
  • T5-XXL text encoder for 4096-dim context embeddings
  • Rectified flow training objective for efficiency
  • Multi-resolution training with masking strategy

Technical specifications:

  • Architecture: STDiT (Spatial-Temporal Diffusion Transformer)
  • Hidden dimension: 1152
  • Transformer layers: 28
  • Attention heads: 16
  • Patch size: 2
  • Latent channels: 4 (3D causal VAE)
  • Context dimension: 4096 (T5-XXL)
  • Training objective: Rectified flow
  • Multi-resolution training with masking
  • Default: 51 frames at 24 FPS (~2.1 seconds)
  • Supports: text-to-video, image-to-video, video-to-video

Reference: Zheng et al., “Open-Sora: Democratizing Efficient Video Production for All”, 2024

Example

var referenceFrame = Tensor<float>.CreateRandom(1, 3, 32, 32);
// Create with industry-standard defaults
var openSora = new OpenSoraModel<float>();
            
// Generate video from text
var video = openSora.GenerateFromText(
    prompt: "A serene lake reflecting autumn foliage with gentle ripples",
    width: 512,
    height: 512,
    numFrames: 51,
    fps: 24,
    numInferenceSteps: 30,
    guidanceScale: 7.0);
            
// Generate video from an image
var animated = openSora.GenerateFromImage(
    inputImage: referenceFrame,
    numFrames: 51,
    numInferenceSteps: 30);

Constructors

Constructor Summary
OpenSoraModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, DiTNoisePredictor<T> dit, TemporalVAE<T> temporalVAE, IConditioningModule<T> conditioner, int defaultNumFrames, int defaultFPS, int? seed) Initializes a new instance of OpenSoraModel with full customization support.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> TemporalVAE { get; }
IVAEModel<T> VAE { get; }
bool SupportsImageToVideo { get; }
bool SupportsTextToVideo { get; }
bool SupportsVideoToVideo { get; }
int LatentChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()