SoraModel<T>

Models & Types · AiDotNet.Diffusion.Video

Sora-architecture model for DiT-based video generation with native spatiotemporal patches.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → VideoDiffusionModelBase<T> → SoraModel<T>

For Beginners

Sora creates videos from text using a world-simulator approach.

How Sora works:

  1. Text is encoded by dual CLIP + T5 encoders into 4096-dim embeddings
  2. Video is compressed by the 3D causal VAE into 16-channel spatiotemporal patches
  3. The DiT processes patches as a sequence (like tokens in a language model)
  4. Full 3D attention captures spatial and temporal relationships simultaneously
  5. Flow matching denoises the video over scheduled timesteps
  6. The causal VAE decodes patches back to variable-duration video

Key characteristics:

  • Native variable duration and resolution (no fixed grid)
  • Trained on video + image data at native aspect ratios
  • Full 3D attention (no factorization) for maximum quality
  • “World simulator” approach for physically plausible generation
  • 150 frames at 24 FPS by default (~6.25 seconds), up to 60s

When to use Sora:

  • Highest-quality video generation
  • Variable-length and multi-resolution content
  • Physical simulation and world modeling
  • Long-duration video generation

Limitations:

  • Proprietary model (API-only access)
  • Very large model with high compute requirements
  • May struggle with complex physical interactions
  • Generation can be slow for long videos

How It Works

Sora by OpenAI uses a Diffusion Transformer (DiT) operating on spatiotemporal patches of video, enabling native variable-duration, resolution, and aspect-ratio video generation. The model treats videos as sequences of spacetime patches, similar to how LLMs process tokens.

Architecture components:

  • DiT backbone with 48 transformer layers and 3072 hidden dimension
  • 24 attention heads with full 3D spatiotemporal attention
  • 3D spatiotemporal patch embeddings (patch size 2)
  • 3D causal VAE with 16 latent channels for spatiotemporal compression
  • 4096-dim text conditioning (CLIP + T5 dual encoder)
  • Flow matching training objective

Technical specifications:

  • Architecture: DiT with 3D spatiotemporal patches
  • Hidden dimension: 3072
  • Transformer layers: 48
  • Attention heads: 24
  • Patch size: 2 (spatiotemporal)
  • Latent channels: 16 (3D causal VAE)
  • Context dimension: 4096 (CLIP + T5 dual encoder)
  • VAE: 3D causal with 3 temporal layers
  • Default: 150 frames at 24 FPS (~6.25 seconds)
  • Training objective: Flow matching
  • Native variable resolution and duration

Reference: OpenAI, “Video generation models as world simulators”, 2024

Example

var referenceFrame = Tensor<float>.CreateRandom(1, 3, 32, 32);
// Create with industry-standard defaults
var sora = new SoraModel<float>();
            
// Generate video from text
var video = sora.GenerateFromText(
    prompt: "A drone shot flying through a dense forest canopy at golden hour",
    width: 1920,
    height: 1080,
    numFrames: 150,
    fps: 24,
    numInferenceSteps: 50,
    guidanceScale: 7.5);
            
// Generate video from an image
var animated = sora.GenerateFromImage(
    inputImage: referenceFrame,
    numFrames: 150,
    numInferenceSteps: 50);

Constructors

Constructor Summary
SoraModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, DiTNoisePredictor<T> dit, TemporalVAE<T> temporalVAE, IConditioningModule<T> conditioner, int defaultNumFrames, int defaultFPS, int? seed) Initializes a new instance of SoraModel with full customization support.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> TemporalVAE { get; }
IVAEModel<T> VAE { get; }
bool SupportsImageToVideo { get; }
bool SupportsTextToVideo { get; }
bool SupportsVideoToVideo { get; }
int LatentChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()