AnimateDiffModel<T>
Models & Types · AiDotNet.Diffusion.Video
AnimateDiff model for text-to-video and image-to-video generation.
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → VideoDiffusionModelBase<T> → AnimateDiffModel<T>
For Beginners
Think of AnimateDiff as “teaching an image generator to make videos.”
How it works:
- Start with a text-to-image model (like Stable Diffusion)
- Add special “motion modules” between the layers
- These modules learn how things move in videos
- The original image quality is preserved while adding motion
Key advantages:
- Works with any Stable Diffusion model/checkpoint
- Can use existing LoRAs, ControlNets, etc.
- Flexible: text-to-video, image-to-video, or both
- Lower training requirements than full video models
Example use cases:
- Generate a short animation from a text prompt
- Animate a still image with natural motion
- Create consistent character animations
- Style transfer for videos using SD checkpoints
How It Works
AnimateDiff extends Stable Diffusion with motion modules that enable temporal consistency in video generation. Unlike SVD which is trained end-to-end for video, AnimateDiff adds motion modules to existing text-to-image models, making it highly flexible.
Architecture overview:
- Base: Standard Stable Diffusion U-Net
- Motion Modules: Temporal attention layers inserted after spatial attention
- VAE: Standard SD VAE (per-frame encoding/decoding)
- Optional: LoRA adapters for style customization
Supported modes:
- Text-to-Video: Generate video from text prompt
- Image-to-Video: Animate an input image with text guidance
- Video-to-Video: Style transfer or modify existing video
Example
// Create AnimateDiff with default motion modules
var animateDiff = new AnimateDiffModel<float>();
// Text-to-video generation
var video = animateDiff.GenerateFromText(
"A beautiful sunset over the ocean, waves gently rolling",
width: 512,
height: 512,
numFrames: 16,
numInferenceSteps: 25);
// Image-to-video with text guidance
var inputImage = ImageHelper<float>.LoadImage("beach.jpg");
var animatedVideo = animateDiff.GenerateFromImage(
inputImage,
numFrames: 16);
Constructors
| Constructor | Summary |
|---|---|
AnimateDiffModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> unet, StandardVAE<T> vae, IConditioningModule<T> conditioner, MotionModuleConfig motionConfig, int defaultNumFrames, int defaultFPS) |
Initializes a new instance of AnimateDiffModel with full customization support. |
Properties
| Property | Summary |
|---|---|
IConditioningModule<T> Conditioner { get; } |
Gets the conditioning module. |
INoisePredictor<T> NoisePredictor { get; } |
Gets the noise predictor. |
IVAEModel<T> VAE { get; } |
Gets the VAE for frame encoding/decoding. |
MotionModuleConfig MotionConfig { get; } |
Gets the motion module configuration. |
bool SupportsImageToVideo { get; } |
Gets whether image-to-video is supported. |
bool SupportsTextToVideo { get; } |
Gets whether text-to-video is supported. |
bool SupportsVideoToVideo { get; } |
Gets whether video-to-video is supported. |
int ContextLength { get; set; } |
Gets or sets the context length for temporal attention. |
int ContextOverlap { get; set; } |
Gets or sets the context overlap for sliding window generation. |
int LatentChannels { get; } |
Gets the number of latent channels. |
Methods
| Method | Summary |
|---|---|
Tensor<T> GenerateFromImage(Tensor<T> inputImage, int? numFrames, int? fps, int numInferenceSteps, int? motionBucketId, double noiseAugStrength, int? seed) |
Generates video from an input image. |
Tensor<T> GenerateFromText(string prompt, string negativePrompt, int width, int height, int? numFrames, int? fps, int numInferenceSteps, double guidanceScale, int? seed) |
Generates video from text using AnimateDiff. |
Fields
| Field | Summary |
|---|---|
int DefaultHeight |
Default AnimateDiff height (SD compatible). |
int DefaultWidth |
Default AnimateDiff width (SD compatible). |