StableVideoDiffusion<T>

Models & Types · AiDotNet.Diffusion.Video

Stable Video Diffusion (SVD) model for image-to-video generation.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → VideoDiffusionModelBase<T> → StableVideoDiffusion<T>

For Beginners

Think of SVD as “making a picture come to life.” You give it a single image, and it generates a short video showing how that scene might animate:

Example workflow:

  1. Input: Photo of a waterfall
  2. SVD analyzes the scene and understands what should move
  3. Output: 4-second video showing water flowing, mist rising

Key features:

  • Image-to-video: Primary use case, animate still images
  • Motion control: Adjust how much motion to add (motion bucket)
  • Configurable length: Generate different numbers of frames
  • High quality: Based on Stable Diffusion’s proven architecture

Compared to text-to-video:

  • More predictable results (scene is defined by input image)
  • Better quality (less ambiguity than text prompts)
  • Faster generation (can use fewer denoising steps)

How It Works

Stable Video Diffusion generates short video clips from a single input image. It extends the Stable Diffusion architecture with temporal awareness, using a 3D U-Net for noise prediction and a temporal VAE for encoding/decoding.

Technical specifications:

  • Default resolution: 576x1024 or 1024x576
  • Default frames: 25 frames at 7 FPS (~3.5 seconds)
  • Motion bucket ID: 1-255 (127 = moderate motion)
  • Noise augmentation: 0.02 default for conditioning image
  • Latent space: 4 channels, 8x spatial downsampling

Example

// Create a Stable Video Diffusion model
var svd = new StableVideoDiffusion<float>();
            
// Load your image (batch=1, channels=3, height=576, width=1024)
var inputImage = ImageHelper<float>.LoadImage("landscape.jpg");
            
// Generate video with default settings
var video = svd.GenerateFromImage(inputImage);
            
// Generate with custom motion (more movement)
var dynamicVideo = svd.GenerateFromImage(
    inputImage,
    numFrames: 25,
    fps: 7,
    motionBucketId: 200,  // Higher = more motion
    numInferenceSteps: 25,
    seed: 42);
            
// Output shape: [1, 25, 3, 576, 1024] (batch, frames, channels, height, width)
// dynamicVideo is the result tensor — write it out with your own encoder.

Constructors

Constructor Summary
StableVideoDiffusion(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, VideoUNetPredictor<T> videoUNet, TemporalVAE<T> temporalVAE, IConditioningModule<T> conditioner, int defaultNumFrames, int defaultFPS, double noiseAugmentStrength) Initializes a new instance of StableVideoDiffusion with default parameters.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; } Gets the conditioning module if available.
INoisePredictor<T> NoisePredictor { get; } Gets the noise predictor used by this model.
IVAEModel<T> TemporalVAE { get; } Gets the temporal VAE specifically for video operations.
IVAEModel<T> VAE { get; } Gets the VAE used by this model for image encoding.
VideoUNetPredictor<T> VideoUNet { get; } Gets the video U-Net predictor with image conditioning support.
bool SupportsImageToVideo { get; } Gets whether this model supports image-to-video generation.
bool SupportsTextToVideo { get; } Gets whether this model supports text-to-video generation.
bool SupportsVideoToVideo { get; } Gets whether this model supports video-to-video transformation.
int LatentChannels { get; } Gets the number of latent channels (4 for SVD).

Methods

Method Summary
Tensor<T> GenerateFromImage(Tensor<T> inputImage, int? numFrames, int? fps, int numInferenceSteps, int? motionBucketId, double noiseAugStrength, int? seed) Generates a video from an input image using image-to-video diffusion.
Tensor<T> GenerateWithEndImageGuidance(Tensor<T> startImage, Tensor<T> endImage, int numFrames, int numInferenceSteps, int? seed) Generates video with motion guidance from a secondary image.
Tensor<T> GenerateWithFirstFrame(Tensor<T> firstFrame, int numFrames, int motionBucketId, int numInferenceSteps, int? seed) Generates video with explicit first frame control.
ValueTuple<int, int> GetRecommendedResolution(double aspectRatio) Gets the recommended resolution for SVD generation.

Fields

Field Summary
int DefaultHeight Default height for SVD generation.
int DefaultWidth Default width for SVD generation.