UpscaleAVideoModel<T>
Models & Types · AiDotNet.Diffusion.SuperResolution
Upscale-A-Video model for temporally consistent video super-resolution with diffusion.
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → VideoDiffusionModelBase<T> → UpscaleAVideoModel<T>
For Beginners
Upscale-A-Video increases video resolution by 4x without flickering.
How Upscale-A-Video works:
- Each frame is encoded with a temporal VAE that considers neighboring frames
- The noised RGB low-resolution video is concatenated with the four-channel latent (7 input channels)
- Temporal attention ensures frames are consistent with each other
- Flow-guided propagation maintains consistency across long sequences
- The temporal VAE decodes the result to high-resolution flicker-free video
Key characteristics:
- 4x video upscaling with temporal consistency
- Temporal attention layers prevent inter-frame flickering
- Flow-guided recurrent propagation for long-range coherence
- Built on SD architecture with temporal extensions
- Processes 16 frames at a time at 24 FPS by default
When to use Upscale-A-Video:
- Upscaling low-resolution video recordings
- Enhancing video quality for streaming/display
- Restoring old or compressed video footage
- Improving AI-generated video resolution
Limitations:
- Fixed 4x upscale factor
- High VRAM requirements due to temporal processing
- Processing speed limited by number of frames per batch
- May introduce subtle artifacts at scene transitions
How It Works
Upscale-A-Video extends image super-resolution to video with temporal consistency, using convolutional temporal modules and flow-guided recurrent propagation to achieve flicker-free 4x upscaling of video content.
Architecture components:
- Video U-Net with released temporal 3D/spatial ResNet modules
- 7 input channels (4 latent + 3-channel noised low-resolution conditioning)
- 9 stage-level convolutional temporal modules plus zero-initialized Transformer3D temporal attention
- Temporal VAE for temporally coherent encoding/decoding
- Flow-guided recurrent propagation for long-range temporal consistency
- DDIM scheduler for efficient inference
Technical specifications:
- Architecture: Video U-Net with temporal 3D CNN modules + temporal VAE
- Input channels: 7 (4 latent noise + 3-channel noised low-resolution video)
- Output channels: 4 (latent space)
- Base channels: 256, multipliers [1, 2, 2, 4]
- Cross-attention dimension: 1024
- Temporal attention layers: 2 per block
- Temporal VAE: 3-frame kernel, 1 temporal layer
- Noise schedule: Scaled linear beta [0.00085, 0.012], 1000 timesteps
- Default frames: 16 at 24 FPS
- Upscale factor: 4x
- Scheduler: DDIM for efficient video inference
Reference: Zhou et al., “Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution”, CVPR 2024
Example
var lowResVideo = Tensor<float>.CreateRandom(1, 3, 32, 32);
var referenceFrame = Tensor<float>.CreateRandom(1, 3, 32, 32);
// Create with industry-standard defaults
var upscaler = new UpscaleAVideoModel<float>();
// Upscale a video (video-to-video super-resolution)
var upscaledVideo = upscaler.VideoToVideo(
inputVideo: lowResVideo,
prompt: "best quality, extremely detailed",
strength: 0.7,
numInferenceSteps: 50,
guidanceScale: 7.5);
// Generate video from an image (animate + upscale)
var videoFromImage = upscaler.GenerateFromImage(
inputImage: referenceFrame,
numFrames: 16,
fps: 24,
numInferenceSteps: 25);
Constructors
| Constructor | Summary |
|---|---|
UpscaleAVideoModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, VideoUNetPredictor<T> videoUNet, TemporalVAE<T> temporalVAE, IConditioningModule<T> conditioner, int defaultNumFrames, int defaultFPS, int? seed) |
Initializes a new instance of UpscaleAVideoModel with full customization support. |
Properties
| Property | Summary |
|---|---|
IConditioningModule<T> Conditioner { get; } |
|
INoisePredictor<T> NoisePredictor { get; } |
|
IVAEModel<T> TemporalVAE { get; } |
|
IVAEModel<T> VAE { get; } |
|
bool SupportsImageToVideo { get; } |
|
bool SupportsTextToVideo { get; } |
|
bool SupportsVideoToVideo { get; } |
|
int LatentChannels { get; } |
|
int UpscaleFactor { get; } |
Gets the video upscale factor (4x). |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> Upscale(Tensor<T> lowResolutionVideo, string prompt, int numInferenceSteps, double guidanceScale, int? seed, int noiseLevel, int temporalWindowSize, int temporalWindowOverlap, Tensor<T> forwardFlows, Tensor<T> backwardFlows, IReadOnlyCollection<int> propagationSteps, string negativePrompt) |
Runs the Upscale-A-Video super-resolution path with a distinct low-resolution conditioning latent for every frame. |
void TrainConditioned(Tensor<T> lowResolutionVideo, Tensor<T> highResolutionVideo, string prompt, int noiseLevel) |
Trains the Upscale-A-Video denoiser on a low/high-resolution video pair. |
Fields
| Field | Summary |
|---|---|
double ReferenceLatentScaleFactor |
Stable Diffusion x4-upscaler VAE scaling factor. |
int DefaultHeight |
Default output video height (576, 4x upscaled from 144). |
int DefaultWidth |
Default output video width (1024, 4x upscaled from 256). |
int ReferenceMaximumNoiseLevel |
Maximum low-resolution noise level used by the reference pipeline. |
int ReferenceWindowOverlap |
Overlap between adjacent temporal denoising windows. |
int ReferenceWindowSize |
Temporal denoising window used by the reference implementation. |