VideoCrafterModel<T>

Models & Types · AiDotNet.Diffusion.Video

VideoCrafter model for high-quality text-to-video and image-to-video generation.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → VideoDiffusionModelBase<T> → VideoCrafterModel<T>

For Beginners

VideoCrafter is like having two video generation modes in one:

Mode 1 - Text-to-Video:

  • Input: “A rocket launching into space”
  • Output: 5-second video of a rocket launch

Mode 2 - Image-to-Video:

  • Input: Photo of a rocket on launch pad
  • Output: 5-second video of the rocket launching

Key advantages:

  • High visual quality (up to 1024x576 resolution)
  • Long video generation (up to 16+ seconds)
  • Good temporal coherence (smooth motion)
  • Dual conditioning (text + image together)

Unlike AnimateDiff which adds motion to SD models, VideoCrafter is trained end-to-end specifically for video generation, resulting in better quality.

How It Works

VideoCrafter is a video generation model that combines the strengths of text-to-video and image-to-video generation. It uses a dual-conditioning approach that enables both modalities while maintaining high visual quality and temporal coherence.

Architecture:

  • 3D U-Net with factorized spatial-temporal attention
  • Dual cross-attention for text and image conditioning
  • Temporal VAE for consistent frame encoding
  • DDIM scheduler for fast inference

Example

// Create VideoCrafter model
var videoCrafter = new VideoCrafterModel<float>();
            
// Text-to-video generation
var video = videoCrafter.GenerateFromText(
    prompt: "A beautiful sunset over the ocean, waves crashing",
    width: 1024,
    height: 576,
    numFrames: 16,
    numInferenceSteps: 50);
            
// Image-to-video with text guidance
var inputImage = ImageHelper<float>.LoadImage("sunset.jpg");
var animatedVideo = videoCrafter.GenerateFromImageAndText(
    image: inputImage,
    prompt: "waves gently rolling, seagulls flying",
    numFrames: 16);

Constructors

Constructor Summary
VideoCrafterModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, VideoUNetPredictor<T> videoUNet, TemporalVAE<T> temporalVAE, IConditioningModule<T> textConditioner, IConditioningModule<T> imageConditioner, int defaultNumFrames, int defaultFPS) Initializes a new instance of VideoCrafterModel with full customization support.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; } Gets the primary conditioning module (text).
IConditioningModule<T> ImageConditioner { get; } Gets the image conditioning module.
INoisePredictor<T> NoisePredictor { get; } Gets the noise predictor.
IVAEModel<T> TemporalVAE { get; } Gets the temporal VAE.
IVAEModel<T> VAE { get; } Gets the VAE.
bool SupportsImageToVideo { get; } Gets whether image-to-video is supported.
bool SupportsTextToVideo { get; } Gets whether text-to-video is supported.
bool SupportsVideoToVideo { get; } Gets whether video-to-video is supported.
bool UseDualConditioning { get; set; } Gets or sets whether to use dual conditioning (text + image together).
double ImageConditioningScale { get; set; } Gets or sets the image conditioning scale.
int LatentChannels { get; } Gets the latent channels.

Methods

Method Summary
Tensor<T> GenerateFromImage(Tensor<T> inputImage, int? numFrames, int? fps, int numInferenceSteps, int? motionBucketId, double noiseAugStrength, int? seed) Generates video from image with optional text guidance.
Tensor<T> GenerateFromImageAndText(Tensor<T> image, string prompt, string negativePrompt, int? numFrames, int numInferenceSteps, double guidanceScale, double imageScale, int? seed) Generates video with dual conditioning (image + text).
Tensor<T> GenerateFromText(string prompt, string negativePrompt, int width, int height, int? numFrames, int? fps, int numInferenceSteps, double guidanceScale, int? seed) Generates video from text prompt.

Fields

Field Summary
int DefaultHeight Default VideoCrafter height.
int DefaultWidth Default VideoCrafter width.