VideoCrafterModel<T>
Models & Types · AiDotNet.Diffusion.Video
VideoCrafter model for high-quality text-to-video and image-to-video generation.
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → VideoDiffusionModelBase<T> → VideoCrafterModel<T>
For Beginners
VideoCrafter is like having two video generation modes in one:
Mode 1 - Text-to-Video:
- Input: “A rocket launching into space”
- Output: 5-second video of a rocket launch
Mode 2 - Image-to-Video:
- Input: Photo of a rocket on launch pad
- Output: 5-second video of the rocket launching
Key advantages:
- High visual quality (up to 1024x576 resolution)
- Long video generation (up to 16+ seconds)
- Good temporal coherence (smooth motion)
- Dual conditioning (text + image together)
Unlike AnimateDiff which adds motion to SD models, VideoCrafter is trained end-to-end specifically for video generation, resulting in better quality.
How It Works
VideoCrafter is a video generation model that combines the strengths of text-to-video and image-to-video generation. It uses a dual-conditioning approach that enables both modalities while maintaining high visual quality and temporal coherence.
Architecture:
- 3D U-Net with factorized spatial-temporal attention
- Dual cross-attention for text and image conditioning
- Temporal VAE for consistent frame encoding
- DDIM scheduler for fast inference
Example
// Create VideoCrafter model
var videoCrafter = new VideoCrafterModel<float>();
// Text-to-video generation
var video = videoCrafter.GenerateFromText(
prompt: "A beautiful sunset over the ocean, waves crashing",
width: 1024,
height: 576,
numFrames: 16,
numInferenceSteps: 50);
// Image-to-video with text guidance
var inputImage = ImageHelper<float>.LoadImage("sunset.jpg");
var animatedVideo = videoCrafter.GenerateFromImageAndText(
image: inputImage,
prompt: "waves gently rolling, seagulls flying",
numFrames: 16);
Constructors
| Constructor | Summary |
|---|---|
VideoCrafterModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, VideoUNetPredictor<T> videoUNet, TemporalVAE<T> temporalVAE, IConditioningModule<T> textConditioner, IConditioningModule<T> imageConditioner, int defaultNumFrames, int defaultFPS) |
Initializes a new instance of VideoCrafterModel with full customization support. |
Properties
| Property | Summary |
|---|---|
IConditioningModule<T> Conditioner { get; } |
Gets the primary conditioning module (text). |
IConditioningModule<T> ImageConditioner { get; } |
Gets the image conditioning module. |
INoisePredictor<T> NoisePredictor { get; } |
Gets the noise predictor. |
IVAEModel<T> TemporalVAE { get; } |
Gets the temporal VAE. |
IVAEModel<T> VAE { get; } |
Gets the VAE. |
bool SupportsImageToVideo { get; } |
Gets whether image-to-video is supported. |
bool SupportsTextToVideo { get; } |
Gets whether text-to-video is supported. |
bool SupportsVideoToVideo { get; } |
Gets whether video-to-video is supported. |
bool UseDualConditioning { get; set; } |
Gets or sets whether to use dual conditioning (text + image together). |
double ImageConditioningScale { get; set; } |
Gets or sets the image conditioning scale. |
int LatentChannels { get; } |
Gets the latent channels. |
Methods
| Method | Summary |
|---|---|
Tensor<T> GenerateFromImage(Tensor<T> inputImage, int? numFrames, int? fps, int numInferenceSteps, int? motionBucketId, double noiseAugStrength, int? seed) |
Generates video from image with optional text guidance. |
Tensor<T> GenerateFromImageAndText(Tensor<T> image, string prompt, string negativePrompt, int? numFrames, int numInferenceSteps, double guidanceScale, double imageScale, int? seed) |
Generates video with dual conditioning (image + text). |
Tensor<T> GenerateFromText(string prompt, string negativePrompt, int width, int height, int? numFrames, int? fps, int numInferenceSteps, double guidanceScale, int? seed) |
Generates video from text prompt. |
Fields
| Field | Summary |
|---|---|
int DefaultHeight |
Default VideoCrafter height. |
int DefaultWidth |
Default VideoCrafter width. |