OpenSoraModel<T>
Models & Types · AiDotNet.Diffusion.Video
Open-Sora model for open-source Sora-like video generation with STDiT architecture.
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → VideoDiffusionModelBase<T> → OpenSoraModel<T>
For Beginners
Open-Sora is an open-source alternative to OpenAI’s Sora.
How Open-Sora works:
- Text prompt is encoded by T5-XXL into 4096-dim embeddings
- Video is compressed by the 3D causal VAE into 4-channel latent space
- The STDiT applies spatial attention first, then temporal attention per layer
- Rectified flow denoises the video latent over scheduled timesteps
- The causal VAE decodes the latent back to video frames
Key characteristics:
- Open-source community project replicating Sora capabilities
- STDiT factorizes spatial and temporal attention for efficiency
- Rectified flow training (straighter trajectories than DDPM)
- Multi-resolution training enables variable aspect ratio/duration
- 51 frames at 24 FPS by default (~2.1 seconds)
When to use Open-Sora:
- Open-source video generation research
- Variable-resolution video generation
- Text-to-video, image-to-video, and video-to-video tasks
- Community-driven experimentation and fine-tuning
Limitations:
- Quality below commercial models (Sora, Veo)
- Research-stage model with ongoing improvements
- Limited duration compared to larger models
- Requires significant compute for training
How It Works
Open-Sora is an open-source reproduction of Sora-like capabilities using the Spatial-Temporal DiT (STDiT) architecture. It features efficient spatial-temporal attention factorization, rectified flow training, and multi-resolution masking strategy.
Architecture components:
- STDiT (Spatial-Temporal Diffusion Transformer) with 28 layers
- 1152 hidden dimension with 16 attention heads
- 3D causal VAE with 4 latent channels for spatiotemporal compression
- T5-XXL text encoder for 4096-dim context embeddings
- Rectified flow training objective for efficiency
- Multi-resolution training with masking strategy
Technical specifications:
- Architecture: STDiT (Spatial-Temporal Diffusion Transformer)
- Hidden dimension: 1152
- Transformer layers: 28
- Attention heads: 16
- Patch size: 2
- Latent channels: 4 (3D causal VAE)
- Context dimension: 4096 (T5-XXL)
- Training objective: Rectified flow
- Multi-resolution training with masking
- Default: 51 frames at 24 FPS (~2.1 seconds)
- Supports: text-to-video, image-to-video, video-to-video
Reference: Zheng et al., “Open-Sora: Democratizing Efficient Video Production for All”, 2024
Example
var referenceFrame = Tensor<float>.CreateRandom(1, 3, 32, 32);
// Create with industry-standard defaults
var openSora = new OpenSoraModel<float>();
// Generate video from text
var video = openSora.GenerateFromText(
prompt: "A serene lake reflecting autumn foliage with gentle ripples",
width: 512,
height: 512,
numFrames: 51,
fps: 24,
numInferenceSteps: 30,
guidanceScale: 7.0);
// Generate video from an image
var animated = openSora.GenerateFromImage(
inputImage: referenceFrame,
numFrames: 51,
numInferenceSteps: 30);
Constructors
| Constructor | Summary |
|---|---|
OpenSoraModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, DiTNoisePredictor<T> dit, TemporalVAE<T> temporalVAE, IConditioningModule<T> conditioner, int defaultNumFrames, int defaultFPS, int? seed) |
Initializes a new instance of OpenSoraModel with full customization support. |
Properties
| Property | Summary |
|---|---|
IConditioningModule<T> Conditioner { get; } |
|
INoisePredictor<T> NoisePredictor { get; } |
|
IVAEModel<T> TemporalVAE { get; } |
|
IVAEModel<T> VAE { get; } |
|
bool SupportsImageToVideo { get; } |
|
bool SupportsTextToVideo { get; } |
|
bool SupportsVideoToVideo { get; } |
|
int LatentChannels { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |