IVideoDiffusionModel<T>

Interfaces · AiDotNet.Interfaces

Interface for video diffusion models that generate temporal sequences.

Implements: IDiffusionModel<T>

Known implementations: VideoDiffusionModelBase<T>

For Beginners

Video diffusion is like image diffusion, but it creates videos instead of single images. The main challenge is making the frames look consistent over time (no flickering or teleporting objects).

How video diffusion works:

  1. The model generates multiple frames at once (typically 14-25 frames)
  2. Special “temporal attention” ensures frames are consistent
  3. The model can be conditioned on a starting image, text, or both

Common approaches:

  • Image-to-Video (SVD): Start from an image, generate motion
  • Text-to-Video (VideoCrafter): Generate video from text description
  • Video-to-Video: Transform existing video with new style/content

Key challenges solved by these models:

  • Temporal consistency (no flickering)
  • Motion coherence (objects move naturally)
  • Long-range dependencies (beginning and end are related)

How It Works

Video diffusion models extend image diffusion to handle the temporal dimension, generating coherent video sequences. They model both spatial (within-frame) and temporal (across-frame) dependencies.

This interface extends IDiffusionModel with video-specific operations.

Properties

Property Summary
bool SupportsImageToVideo { get; } Gets whether this model supports image-to-video generation.
bool SupportsTextToVideo { get; } Gets whether this model supports text-to-video generation.
bool SupportsVideoToVideo { get; } Gets whether this model supports video-to-video transformation.
int DefaultFPS { get; } Gets the default frames per second for generated videos.
int DefaultNumFrames { get; } Gets the default number of frames generated.
int MotionBucketId { get; } Gets the motion bucket ID for controlling motion intensity (SVD-specific).

Methods

Method Summary
Tensor<T> ExtractFrame(Tensor<T> video, int frameIndex) Extracts a frame from the video tensor.
Tensor<T> FramesToVideo(Tensor<T>[] frames) Concatenates frames into a video tensor.
Tensor<T> GenerateFromImage(Tensor<T> inputImage, int? numFrames, int? fps, int numInferenceSteps, int? motionBucketId, double noiseAugStrength, int? seed) Generates a video from a conditioning image.
Tensor<T> GenerateFromText(string prompt, string negativePrompt, int width, int height, int? numFrames, int? fps, int numInferenceSteps, double guidanceScale, int? seed) Generates a video from a text prompt.
Tensor<T> InterpolateFrames(Tensor<T> video, int targetFPS, FrameInterpolationMethod interpolationMethod) Interpolates between frames to increase frame rate.
Tensor<T> VideoToVideo(Tensor<T> inputVideo, string prompt, string negativePrompt, double strength, int numInferenceSteps, double guidanceScale, int? seed) Transforms an existing video.
void SetMotionBucketId(int bucketId) Sets the motion intensity for generation.