DualXVSR<T>
Models & Types · AiDotNet.Video.Enhancement
DualX-VSR: a dual axial spatial x temporal transformer for real-world video super-resolution without motion compensation.
Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoSuperResolutionBase<T> → DualXVSR<T>
For Beginners
Most video upscalers first work out how things moved between frames. DualX-VSR skips that: its attention looks along a column across all frames, then along a row across all frames, so every position can draw on matching content anywhere in the clip, like solving a crossword by reading down and across.
How It Works
The network is Cao et al.’s DualX-VSR (arXiv 2506.04830, 2025), a ViViT-based model with no optical flow, alignment or recurrence. Each frame passes a 3x3 convolution to 64 channels, a two-block temporal transformer mixes every pixel’s frames, and 2x2 patches become 1,280-wide tokens. Six vertical-temporal blocks attend over all height x time positions of each token column, then six horizontal-temporal blocks over all width x time positions of each row, with rotary positions (RoPE, base 10,000). The tokens are decoded back to pixels, a second two-block temporal transformer runs, and per-frame convolution and pixel shuffle upsample by four.
The paper releases no code, so what it leaves open follows the ViViT implementation it builds on (vit-pytorch): pre-norm residual blocks with LayerNorm, bias-free q/k/v projections to 12 heads of 64 channels, GELU MLPs, a final norm per transformer and the LayerNorm-Linear-LayerNorm patch embedding. With these choices the default network has 127.8M parameters against the paper’s 127.95M. RoPE over a height (or width) x time sequence gives each axis half of a head’s channels; the upsampler is BasicVSR’s (pixel shuffle x2 stages with LeakyReLU, conv_hr, conv_last).
Training follows Sec. 5.1 and App. C.1: AdamW at 5e-5 on Charbonnier loss for 500,000 pretraining and 150,000 fine-tuning iterations (bicubic, then RealBasicVSR’s first-order degradations: data preparation, not the model), then 25,000 iterations with Charbonnier, perceptual and GAN losses weighted 0.01, 1 and 0.005 following BasicVSR, at 1e-5 for the generator and RealBasicVSR’s spectral-norm U-Net discriminator, with its VGG19 perceptual loss and vanilla GAN loss. AdamW’s betas and decay are unstated, so they are PyTorch’s defaults.
Example
var arch = new NeuralNetworkArchitecture<float>(inputType: InputType.FourDimensional,
taskType: NeuralNetworkTaskType.Regression, inputFrames: 16, inputHeight: 64, inputWidth: 64, inputDepth: 3);
var model = new DualXVSR<float>(arch, new DualXVSROptions());
model.Train(degradedClip, cleanClip); // [16, 3, 64, 64] -> [16, 3, 256, 256], pixel values 0-255
var upscaled = model.Upscale(degradedClip);
Constructors
| Constructor | Summary |
|---|---|
DualXVSR(NeuralNetworkArchitecture<T> architecture, DualXVSROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a DualX-VSR model in native training mode. |
DualXVSR(NeuralNetworkArchitecture<T> architecture, string modelPath, DualXVSROptions options) |
Creates a DualX-VSR model in native training mode. |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
Frames are pixels on the 0-255 scale; the network divides them by 255. |
LayerInputDomain GetOutputDomain(int[] outputShape) |
Upscaled frames come back on the same 0-255 scale. |
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Tensor<T> Upscale(Tensor<T> lowResFrames) |