DOVE<T>

Models & Types · AiDotNet.Video.Enhancement

DOVE: efficient one-step diffusion for real-world video super-resolution, fine-tuned from CogVideoX1.5-5B.

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoSuperResolutionBase<T> → DOVE<T>

For Beginners

DOVE enlarges each frame the simple way and then lets a large video generation model clean the enlargement up in a single pass. Running one step instead of dozens makes it many times faster than other diffusion upscalers.

How It Works

DOVE (Chen et al., NeurIPS 2025) adds no module to CogVideoX1.5: the clip’s bilinear x4 upscale is encoded by the causal 3D VAE and taken as the noisy latent at timestep 399, and one v-prediction step of the expert transformer, z_sr = sqrt(ᾱ_t)·z_lr − sqrt(1 − ᾱ_t)·v_θ(z_lr, c, t), gives the latent the VAE decodes (Eq. 2). The prompt is empty and pre-encoded. Only the transformer trains: first on the MSE between z_sr and the high-resolution latent (Eq. 3), then in pixel space with the VAE run frame by frame, on MSE, DISTS and the frame-difference loss, mostly on single frames (Eqs. 4-7). The release (zhengchen1999/DOVE) fixes the remaining details: frames padded to 8N + 1 by repeating the last, sides to multiples of 16, the posterior sampled, and the first latent frame repeated to fill the temporal patch.

The empty prompt’s T5-XXL embedding is part of the checkpoint (the release ships it pre-encoded) and is held as a frozen tensor; untrained, it is zero.

Constructors

Constructor Summary
DOVE(NeuralNetworkArchitecture<T> architecture, DOVEOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a DOVE model in native training mode.
DOVE(NeuralNetworkArchitecture<T> architecture, string modelPath, DOVEOptions options) Creates a DOVE model in native training mode.

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape) Frames are pixels on the 0-255 scale; the network divides them by 255.
LayerInputDomain GetOutputDomain(int[] outputShape) Upscaled frames come back on the same 0-255 scale.
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> Upscale(Tensor<T> lowResFrames)