MakeAVideoModel<T>
Models & Types · AiDotNet.Diffusion.Video
Make-A-Video model — text-to-video generation without paired text-video data.
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → VideoDiffusionModelBase<T> → MakeAVideoModel<T>
For Beginners
Make-A-Video is Meta’s video generation model that creates videos from text descriptions without needing paired text-video training data.
How Make-A-Video works:
- Text prompt is encoded using CLIP and BPE into a 768-dimensional embedding
- A pseudo-3D U-Net generates an initial low-resolution video in latent space
- Temporal layers extend single images into coherent video sequences
- Spatial and temporal super-resolution stages increase quality and frame rate
Advantages:
- Does not require paired text-video training data
- Leverages existing high-quality text-to-image knowledge
- Supports both text-to-video and image-to-video generation
- Pseudo-3D convolutions are more memory-efficient than full 3D
Limitations:
- Lower temporal consistency than full 3D attention models
- Limited to shorter video clips (16 frames default)
- Quality depends heavily on the underlying text-to-image model
How It Works
Make-A-Video leverages text-to-image models and unsupervised video learning to generate videos without requiring paired text-video training data. It uses a three-stage pipeline: text-to-image base generation, temporal extension for motion, and spatial plus temporal super-resolution for high-quality output.
Architecture components:
- Pseudo-3D U-Net with temporal convolutions and attention
- Standard VAE for spatial latent encoding (4-channel latent space)
- CLIP + BPE dual text encoder for conditioning (768-dim context)
- Three-stage cascade: base T2I, temporal extension, spatial/temporal SR
Technical specifications:
- Architecture: Pseudo-3D U-Net with temporal conv and attention
- Latent space: 4 channels with standard VAE
- Text encoder: CLIP + BPE (768-dimensional context)
- Base resolution: 64x64 latent (256x256 pixel equivalent)
- Super-resolution: up to 768x768 pixels
- Default: 16 frames at 8 FPS
- Noise schedule: DDPM linear
Reference: Singer et al., “Make-A-Video: Text-to-Video Generation without Text-Video Data”, ICLR 2023
Example
var referenceImage = Tensor<float>.CreateRandom(1, 3, 32, 32);
// Create with default settings
var makeAVideo = new MakeAVideoModel<float>();
// Generate video from text
var frames = makeAVideo.GenerateFromText(
prompt: "A teddy bear painting a portrait",
numFrames: 16,
numInferenceSteps: 50,
guidanceScale: 7.5);
// Generate video from an image
var animatedFrames = makeAVideo.GenerateFromImage(
inputImage: referenceImage,
numFrames: 16);
Constructors
| Constructor | Summary |
|---|---|
MakeAVideoModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, VideoUNetPredictor<T> videoUNet, StandardVAE<T> vae, IConditioningModule<T> conditioner, int defaultNumFrames, int defaultFPS, int? seed) |
Initializes a new instance of MakeAVideoModel with full customization support. |
Properties
| Property | Summary |
|---|---|
IConditioningModule<T> Conditioner { get; } |
|
INoisePredictor<T> NoisePredictor { get; } |
|
IVAEModel<T> VAE { get; } |
|
bool SupportsImageToVideo { get; } |
|
bool SupportsTextToVideo { get; } |
|
bool SupportsVideoToVideo { get; } |
|
int LatentChannels { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |