AnimateDiffModel<T>

Models & Types · AiDotNet.Diffusion.Video

AnimateDiff model for text-to-video and image-to-video generation.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → VideoDiffusionModelBase<T> → AnimateDiffModel<T>

For Beginners

Think of AnimateDiff as “teaching an image generator to make videos.”

How it works:

  1. Start with a text-to-image model (like Stable Diffusion)
  2. Add special “motion modules” between the layers
  3. These modules learn how things move in videos
  4. The original image quality is preserved while adding motion

Key advantages:

  • Works with any Stable Diffusion model/checkpoint
  • Can use existing LoRAs, ControlNets, etc.
  • Flexible: text-to-video, image-to-video, or both
  • Lower training requirements than full video models

Example use cases:

  • Generate a short animation from a text prompt
  • Animate a still image with natural motion
  • Create consistent character animations
  • Style transfer for videos using SD checkpoints

How It Works

AnimateDiff extends Stable Diffusion with motion modules that enable temporal consistency in video generation. Unlike SVD which is trained end-to-end for video, AnimateDiff adds motion modules to existing text-to-image models, making it highly flexible.

Architecture overview:

  • Base: Standard Stable Diffusion U-Net
  • Motion Modules: Temporal attention layers inserted after spatial attention
  • VAE: Standard SD VAE (per-frame encoding/decoding)
  • Optional: LoRA adapters for style customization

Supported modes:

  • Text-to-Video: Generate video from text prompt
  • Image-to-Video: Animate an input image with text guidance
  • Video-to-Video: Style transfer or modify existing video

Example

// Create AnimateDiff with default motion modules
var animateDiff = new AnimateDiffModel<float>();
            
// Text-to-video generation
var video = animateDiff.GenerateFromText(
    "A beautiful sunset over the ocean, waves gently rolling",
    width: 512,
    height: 512,
    numFrames: 16,
    numInferenceSteps: 25);
            
// Image-to-video with text guidance
var inputImage = ImageHelper<float>.LoadImage("beach.jpg");
var animatedVideo = animateDiff.GenerateFromImage(
    inputImage,
    numFrames: 16);

Constructors

Constructor Summary
AnimateDiffModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> unet, StandardVAE<T> vae, IConditioningModule<T> conditioner, MotionModuleConfig motionConfig, int defaultNumFrames, int defaultFPS) Initializes a new instance of AnimateDiffModel with full customization support.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; } Gets the conditioning module.
INoisePredictor<T> NoisePredictor { get; } Gets the noise predictor.
IVAEModel<T> VAE { get; } Gets the VAE for frame encoding/decoding.
MotionModuleConfig MotionConfig { get; } Gets the motion module configuration.
bool SupportsImageToVideo { get; } Gets whether image-to-video is supported.
bool SupportsTextToVideo { get; } Gets whether text-to-video is supported.
bool SupportsVideoToVideo { get; } Gets whether video-to-video is supported.
int ContextLength { get; set; } Gets or sets the context length for temporal attention.
int ContextOverlap { get; set; } Gets or sets the context overlap for sliding window generation.
int LatentChannels { get; } Gets the number of latent channels.

Methods

Method Summary
Tensor<T> GenerateFromImage(Tensor<T> inputImage, int? numFrames, int? fps, int numInferenceSteps, int? motionBucketId, double noiseAugStrength, int? seed) Generates video from an input image.
Tensor<T> GenerateFromText(string prompt, string negativePrompt, int width, int height, int? numFrames, int? fps, int numInferenceSteps, double guidanceScale, int? seed) Generates video from text using AnimateDiff.

Fields

Field Summary
int DefaultHeight Default AnimateDiff height (SD compatible).
int DefaultWidth Default AnimateDiff width (SD compatible).