Upscale4KAgent<T>
Models & Types · AiDotNet.Video.Enhancement
4KAgent (Zuo et al. 2025): an agent that upscales any image to 4K by planning super-resolution steps, running every expert in its toolbox on each step, keeping the result a quality score prefers, and rolling back when no result passes; here over AiDotNet’s own video super-resolution networks.
Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoSuperResolutionBase<T> → Upscale4KAgent<T>
For Beginners
instead of one upscaling network, the agent runs several, measures which output looks most natural, keeps that one, and repeats until the clip is large enough.
How It Works
The loop follows the release (taco-group/4KAgent, pipeline/the4kagent_pipeline.py):
- Profile.
Upscale4K,ScaleFactor
and RestorePreference are the profile’s Upscale4K, ScaleFactor and
RestorePerference; the defaults are Gen4K-P.
- Plan (propose, get_target_factor). The factor is the smallest of 2, 4, 8 and 16 that brings the longer
side to 4000 pixels in 4K mode (1 when it is already there), or the profile’s scale factor otherwise; 2 plans one 2x step, 4 one 4x step, 8 a 2x then a 4x step, 16 two 4x steps.
- Quality-driven mixture of experts (execute_subtask). Every expert admitted by the preference upscales
the clip; each result is scored and the best kept. A best score below
RollbackThreshold marks the step a failure.
- Rollback (roll_back, reschedule). A failed step backtracks to try another order of the remaining
subtasks. A plan of super-resolution steps has no other order, so every node is fully expanded and the release compromises by continuing from the best descendant, the result the step already kept.
- Color fix. A super-resolution result whose longer side reaches 1024 pixels takes its input’s
per-channel mean and standard deviation (adain_color_fix).
Native substitutions, each because the release’s component is a hosted model AiDotNet does not carry: the experts are BasicVSR and IconVSR (fidelity) and RealBasicVSR and RealViformer (perception) in place of the release’s image toolbox (SwinIR, HAT, DiffBIR, OSEDiff, …); a 2x step is a 4x expert followed by MATLAB’s antialiased bicubic halving, since every native expert is 4x; the reflection score is the release’s NIQE term, 1 − NIQE / 10, averaged over the frames, in place of HPSv2 plus the mean of CLIPIQA+, MANIQA, MUSIQ and that term; the perception agent is absent because super-resolution is the only subtask the native toolbox serves.
Training trains the toolbox: each expert takes a step on its own paper’s first-stage loss with its own paper’s optimizer against the 4x target. A target at a larger multiple of 4 is area-averaged down to 4x.
Constructors
| Constructor | Summary |
|---|---|
Upscale4KAgent(NeuralNetworkArchitecture<T> architecture, Upscale4KAgentOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a 4KAgent over its native experts. |
Upscale4KAgent(NeuralNetworkArchitecture<T> architecture, string modelPath, Upscale4KAgentOptions options) |
Creates a 4KAgent over its native experts. |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
Frames are pixels on the 0-255 scale; the agent divides them by 255. |
LayerInputDomain GetOutputDomain(int[] outputShape) |
Upscaled frames come back on the same 0-255 scale. |
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Tensor<T> Upscale(Tensor<T> lowResFrames) |