UpscaleAVideoModel<T>

Models & Types · AiDotNet.Diffusion.SuperResolution

Upscale-A-Video model for temporally consistent video super-resolution with diffusion.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → VideoDiffusionModelBase<T> → UpscaleAVideoModel<T>

For Beginners

Upscale-A-Video increases video resolution by 4x without flickering.

How Upscale-A-Video works:

  1. Each frame is encoded with a temporal VAE that considers neighboring frames
  2. The noised RGB low-resolution video is concatenated with the four-channel latent (7 input channels)
  3. Temporal attention ensures frames are consistent with each other
  4. Flow-guided propagation maintains consistency across long sequences
  5. The temporal VAE decodes the result to high-resolution flicker-free video

Key characteristics:

  • 4x video upscaling with temporal consistency
  • Temporal attention layers prevent inter-frame flickering
  • Flow-guided recurrent propagation for long-range coherence
  • Built on SD architecture with temporal extensions
  • Processes 16 frames at a time at 24 FPS by default

When to use Upscale-A-Video:

  • Upscaling low-resolution video recordings
  • Enhancing video quality for streaming/display
  • Restoring old or compressed video footage
  • Improving AI-generated video resolution

Limitations:

  • Fixed 4x upscale factor
  • High VRAM requirements due to temporal processing
  • Processing speed limited by number of frames per batch
  • May introduce subtle artifacts at scene transitions

How It Works

Upscale-A-Video extends image super-resolution to video with temporal consistency, using convolutional temporal modules and flow-guided recurrent propagation to achieve flicker-free 4x upscaling of video content.

Architecture components:

  • Video U-Net with released temporal 3D/spatial ResNet modules
  • 7 input channels (4 latent + 3-channel noised low-resolution conditioning)
  • 9 stage-level convolutional temporal modules plus zero-initialized Transformer3D temporal attention
  • Temporal VAE for temporally coherent encoding/decoding
  • Flow-guided recurrent propagation for long-range temporal consistency
  • DDIM scheduler for efficient inference

Technical specifications:

  • Architecture: Video U-Net with temporal 3D CNN modules + temporal VAE
  • Input channels: 7 (4 latent noise + 3-channel noised low-resolution video)
  • Output channels: 4 (latent space)
  • Base channels: 256, multipliers [1, 2, 2, 4]
  • Cross-attention dimension: 1024
  • Temporal attention layers: 2 per block
  • Temporal VAE: 3-frame kernel, 1 temporal layer
  • Noise schedule: Scaled linear beta [0.00085, 0.012], 1000 timesteps
  • Default frames: 16 at 24 FPS
  • Upscale factor: 4x
  • Scheduler: DDIM for efficient video inference

Reference: Zhou et al., “Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution”, CVPR 2024

Example

var lowResVideo = Tensor<float>.CreateRandom(1, 3, 32, 32);
var referenceFrame = Tensor<float>.CreateRandom(1, 3, 32, 32);
// Create with industry-standard defaults
var upscaler = new UpscaleAVideoModel<float>();
            
// Upscale a video (video-to-video super-resolution)
var upscaledVideo = upscaler.VideoToVideo(
    inputVideo: lowResVideo,
    prompt: "best quality, extremely detailed",
    strength: 0.7,
    numInferenceSteps: 50,
    guidanceScale: 7.5);
            
// Generate video from an image (animate + upscale)
var videoFromImage = upscaler.GenerateFromImage(
    inputImage: referenceFrame,
    numFrames: 16,
    fps: 24,
    numInferenceSteps: 25);

Constructors

Constructor Summary
UpscaleAVideoModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, VideoUNetPredictor<T> videoUNet, TemporalVAE<T> temporalVAE, IConditioningModule<T> conditioner, int defaultNumFrames, int defaultFPS, int? seed) Initializes a new instance of UpscaleAVideoModel with full customization support.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> TemporalVAE { get; }
IVAEModel<T> VAE { get; }
bool SupportsImageToVideo { get; }
bool SupportsTextToVideo { get; }
bool SupportsVideoToVideo { get; }
int LatentChannels { get; }
int UpscaleFactor { get; } Gets the video upscale factor (4x).

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> Upscale(Tensor<T> lowResolutionVideo, string prompt, int numInferenceSteps, double guidanceScale, int? seed, int noiseLevel, int temporalWindowSize, int temporalWindowOverlap, Tensor<T> forwardFlows, Tensor<T> backwardFlows, IReadOnlyCollection<int> propagationSteps, string negativePrompt) Runs the Upscale-A-Video super-resolution path with a distinct low-resolution conditioning latent for every frame.
void TrainConditioned(Tensor<T> lowResolutionVideo, Tensor<T> highResolutionVideo, string prompt, int noiseLevel) Trains the Upscale-A-Video denoiser on a low/high-resolution video pair.

Fields

Field Summary
double ReferenceLatentScaleFactor Stable Diffusion x4-upscaler VAE scaling factor.
int DefaultHeight Default output video height (576, 4x upscaled from 144).
int DefaultWidth Default output video width (1024, 4x upscaled from 256).
int ReferenceMaximumNoiseLevel Maximum low-resolution noise level used by the reference pipeline.
int ReferenceWindowOverlap Overlap between adjacent temporal denoising windows.
int ReferenceWindowSize Temporal denoising window used by the reference implementation.