RunwayGenModel<T>
Models & Types · AiDotNet.Diffusion.Video
Runway Gen model for multi-modal video generation with structure and content disentanglement.
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → VideoDiffusionModelBase<T> → RunwayGenModel<T>
For Beginners
Runway Gen creates professional-quality videos from text and images.
How Runway Gen works:
- Text/image input is encoded by CLIP (Gen-2) or dual encoder (Gen-3)
- Video is compressed by the temporal VAE into 4-channel latent space
- The Video U-Net denoises with temporal attention for frame consistency
- Multi-modal conditioning guides structure and content separately
- The temporal VAE decodes the latent back to video frames
Key characteristics:
- Gen-2: CLIP-based, 320-channel U-Net, 25 frames at 24 FPS
- Gen-3: Enhanced dual encoder, 384-channel U-Net, 150 frames, causal VAE
- Multi-modal: text, image, video, and motion conditioning
- Structure-content disentanglement for precise editing
- Cascaded generation for high resolution output
When to use Runway Gen:
- Professional video generation and editing
- Multi-modal conditioning (text + image + video)
- Video-to-video style transfer
- Image animation with motion control
Limitations:
- Commercial/proprietary model (API-only access)
- Expensive generation costs
- Limited control over internal architecture
- Gen-3 requires significant compute resources
How It Works
Runway Gen (Gen-1/Gen-2/Gen-3) uses temporal diffusion with multi-modal conditioning for photorealistic video generation and editing. The model supports text, image, video, and motion conditioning with structure-content disentanglement.
Architecture components:
- Video U-Net with temporal attention and cross-frame consistency
- Gen-2: 320 base channels, 1024-dim CLIP, 8 heads, 1 temporal layer
- Gen-3: 384 base channels, 2048-dim dual encoder, 16 heads, 3 temporal layers
- Temporal VAE for inter-frame coherence (causal in Gen-3)
- Multi-modal conditioning: text, image, video, and motion
- Structure and style disentanglement for editing control
Technical specifications:
- Architecture: Video U-Net with temporal attention and multi-modal conditioning
- Gen-2: 320 base channels, 1024 cross-attention, 8 heads, 1 temporal layer
- Gen-3: 384 base channels, 2048 cross-attention, 16 heads, 3 temporal layers
- Latent channels: 4 (temporal VAE)
- Channel multipliers: [1, 2, 4, 4]
- Gen-2 default: 25 frames at 24 FPS (~1 second)
- Gen-3 default: 150 frames at 24 FPS (~6.25 seconds)
- Scheduler: DDIM for efficient inference
- Supports: text-to-video, image-to-video, video-to-video
Reference: Esser et al., “Structure and Content-Guided Video Synthesis with Diffusion Models”, ICCV 2023
Example
var referenceFrame = Tensor<float>.CreateRandom(1, 3, 32, 32);
// Create Gen-2 with defaults
var runwayGen2 = new RunwayGenModel<float>();
// Create Gen-3 Alpha variant
var runwayGen3 = RunwayGenModel<float>.CreateGen3Alpha();
// Generate video from text
var video = runwayGen2.GenerateFromText(
prompt: "A cinematic shot of a castle on a cliff at sunset",
width: 1280,
height: 768,
numFrames: 25,
fps: 24,
numInferenceSteps: 50,
guidanceScale: 12.0);
// Generate video from an image
var animated = runwayGen2.GenerateFromImage(
inputImage: referenceFrame,
numFrames: 25,
numInferenceSteps: 50);
Constructors
| Constructor | Summary |
|---|---|
RunwayGenModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, VideoUNetPredictor<T> videoUNet, TemporalVAE<T> temporalVAE, IConditioningModule<T> conditioner, bool isGen3, int defaultNumFrames, int defaultFPS, int? seed) |
Initializes a new instance of RunwayGenModel with full customization support. |
Properties
| Property | Summary |
|---|---|
IConditioningModule<T> Conditioner { get; } |
|
INoisePredictor<T> NoisePredictor { get; } |
|
IVAEModel<T> TemporalVAE { get; } |
|
IVAEModel<T> VAE { get; } |
|
bool IsGen3 { get; } |
Gets whether this is a Gen-3 variant with enhanced architecture. |
bool SupportsImageToVideo { get; } |
|
bool SupportsTextToVideo { get; } |
|
bool SupportsVideoToVideo { get; } |
|
int LatentChannels { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
RunwayGenModel<T> CreateGen3Alpha(IConditioningModule<T> conditioner) |
Creates a Gen-3 Alpha variant with enhanced architecture. |