MakeAVideoModel<T>

Models & Types · AiDotNet.Diffusion.Video

Make-A-Video model — text-to-video generation without paired text-video data.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → VideoDiffusionModelBase<T> → MakeAVideoModel<T>

For Beginners

Make-A-Video is Meta’s video generation model that creates videos from text descriptions without needing paired text-video training data.

How Make-A-Video works:

  1. Text prompt is encoded using CLIP and BPE into a 768-dimensional embedding
  2. A pseudo-3D U-Net generates an initial low-resolution video in latent space
  3. Temporal layers extend single images into coherent video sequences
  4. Spatial and temporal super-resolution stages increase quality and frame rate

Advantages:

  • Does not require paired text-video training data
  • Leverages existing high-quality text-to-image knowledge
  • Supports both text-to-video and image-to-video generation
  • Pseudo-3D convolutions are more memory-efficient than full 3D

Limitations:

  • Lower temporal consistency than full 3D attention models
  • Limited to shorter video clips (16 frames default)
  • Quality depends heavily on the underlying text-to-image model

How It Works

Make-A-Video leverages text-to-image models and unsupervised video learning to generate videos without requiring paired text-video training data. It uses a three-stage pipeline: text-to-image base generation, temporal extension for motion, and spatial plus temporal super-resolution for high-quality output.

Architecture components:

  • Pseudo-3D U-Net with temporal convolutions and attention
  • Standard VAE for spatial latent encoding (4-channel latent space)
  • CLIP + BPE dual text encoder for conditioning (768-dim context)
  • Three-stage cascade: base T2I, temporal extension, spatial/temporal SR

Technical specifications:

  • Architecture: Pseudo-3D U-Net with temporal conv and attention
  • Latent space: 4 channels with standard VAE
  • Text encoder: CLIP + BPE (768-dimensional context)
  • Base resolution: 64x64 latent (256x256 pixel equivalent)
  • Super-resolution: up to 768x768 pixels
  • Default: 16 frames at 8 FPS
  • Noise schedule: DDPM linear

Reference: Singer et al., “Make-A-Video: Text-to-Video Generation without Text-Video Data”, ICLR 2023

Example

var referenceImage = Tensor<float>.CreateRandom(1, 3, 32, 32);
// Create with default settings
var makeAVideo = new MakeAVideoModel<float>();
            
// Generate video from text
var frames = makeAVideo.GenerateFromText(
    prompt: "A teddy bear painting a portrait",
    numFrames: 16,
    numInferenceSteps: 50,
    guidanceScale: 7.5);
            
// Generate video from an image
var animatedFrames = makeAVideo.GenerateFromImage(
    inputImage: referenceImage,
    numFrames: 16);

Constructors

Constructor Summary
MakeAVideoModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, VideoUNetPredictor<T> videoUNet, StandardVAE<T> vae, IConditioningModule<T> conditioner, int defaultNumFrames, int defaultFPS, int? seed) Initializes a new instance of MakeAVideoModel with full customization support.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> VAE { get; }
bool SupportsImageToVideo { get; }
bool SupportsTextToVideo { get; }
bool SupportsVideoToVideo { get; }
int LatentChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()