SoraModel<T>
Models & Types · AiDotNet.Diffusion.Video
Sora-architecture model for DiT-based video generation with native spatiotemporal patches.
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → VideoDiffusionModelBase<T> → SoraModel<T>
For Beginners
Sora creates videos from text using a world-simulator approach.
How Sora works:
- Text is encoded by dual CLIP + T5 encoders into 4096-dim embeddings
- Video is compressed by the 3D causal VAE into 16-channel spatiotemporal patches
- The DiT processes patches as a sequence (like tokens in a language model)
- Full 3D attention captures spatial and temporal relationships simultaneously
- Flow matching denoises the video over scheduled timesteps
- The causal VAE decodes patches back to variable-duration video
Key characteristics:
- Native variable duration and resolution (no fixed grid)
- Trained on video + image data at native aspect ratios
- Full 3D attention (no factorization) for maximum quality
- “World simulator” approach for physically plausible generation
- 150 frames at 24 FPS by default (~6.25 seconds), up to 60s
When to use Sora:
- Highest-quality video generation
- Variable-length and multi-resolution content
- Physical simulation and world modeling
- Long-duration video generation
Limitations:
- Proprietary model (API-only access)
- Very large model with high compute requirements
- May struggle with complex physical interactions
- Generation can be slow for long videos
How It Works
Sora by OpenAI uses a Diffusion Transformer (DiT) operating on spatiotemporal patches of video, enabling native variable-duration, resolution, and aspect-ratio video generation. The model treats videos as sequences of spacetime patches, similar to how LLMs process tokens.
Architecture components:
- DiT backbone with 48 transformer layers and 3072 hidden dimension
- 24 attention heads with full 3D spatiotemporal attention
- 3D spatiotemporal patch embeddings (patch size 2)
- 3D causal VAE with 16 latent channels for spatiotemporal compression
- 4096-dim text conditioning (CLIP + T5 dual encoder)
- Flow matching training objective
Technical specifications:
- Architecture: DiT with 3D spatiotemporal patches
- Hidden dimension: 3072
- Transformer layers: 48
- Attention heads: 24
- Patch size: 2 (spatiotemporal)
- Latent channels: 16 (3D causal VAE)
- Context dimension: 4096 (CLIP + T5 dual encoder)
- VAE: 3D causal with 3 temporal layers
- Default: 150 frames at 24 FPS (~6.25 seconds)
- Training objective: Flow matching
- Native variable resolution and duration
Reference: OpenAI, “Video generation models as world simulators”, 2024
Example
var referenceFrame = Tensor<float>.CreateRandom(1, 3, 32, 32);
// Create with industry-standard defaults
var sora = new SoraModel<float>();
// Generate video from text
var video = sora.GenerateFromText(
prompt: "A drone shot flying through a dense forest canopy at golden hour",
width: 1920,
height: 1080,
numFrames: 150,
fps: 24,
numInferenceSteps: 50,
guidanceScale: 7.5);
// Generate video from an image
var animated = sora.GenerateFromImage(
inputImage: referenceFrame,
numFrames: 150,
numInferenceSteps: 50);
Constructors
| Constructor | Summary |
|---|---|
SoraModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, DiTNoisePredictor<T> dit, TemporalVAE<T> temporalVAE, IConditioningModule<T> conditioner, int defaultNumFrames, int defaultFPS, int? seed) |
Initializes a new instance of SoraModel with full customization support. |
Properties
| Property | Summary |
|---|---|
IConditioningModule<T> Conditioner { get; } |
|
INoisePredictor<T> NoisePredictor { get; } |
|
IVAEModel<T> TemporalVAE { get; } |
|
IVAEModel<T> VAE { get; } |
|
bool SupportsImageToVideo { get; } |
|
bool SupportsTextToVideo { get; } |
|
bool SupportsVideoToVideo { get; } |
|
int LatentChannels { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |