IconVSR<T>
Models & Types · AiDotNet.Video.Enhancement
IconVSR: information-aggregation with coupled propagation for video super-resolution.
Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoSuperResolutionBase<T> → IconVSR<T>
For Beginners
In long videos, processing frames one-by-one can accumulate errors. IconVSR solves this by picking “keyframes” as reliable reference points and letting the forward/backward passes share information, like having checkpoints in a relay race.
How It Works
The network is the authors’ IconVSRNet (open-mmlab/mmagic, models/editors/iconvsr): BasicVSR’s bidirectional recurrent propagation with SPyNet flow alignment, plus information refill and coupled propagation. At keyframes (every fifth frame and the last) the propagated feature is fused by a 3x3 convolution with features an EDVR-M extractor (PCD deformable alignment and TSA fusion) computes from the five surrounding frames, which stops alignment errors from accumulating over long sequences; the forward branch also takes the backward branch’s feature at each frame.
Training follows configs/iconvsr/iconvsr_2xb4_reds4.py: Charbonnier loss, Adam (0.9, 0.99) at 2e-4 with SPyNet at 0.125 times the rate, SPyNet and EDVR frozen for the first 5,000 iterations, cosine annealing to 1e-7. The release loads SPyNet and EDVR-M pretrained; here they start from their release initializations.
Example
var arch = new NeuralNetworkArchitecture<float>(inputType: InputType.FourDimensional,
taskType: NeuralNetworkTaskType.Regression, inputFrames: 15, inputHeight: 64, inputWidth: 64, inputDepth: 3);
var model = new IconVSR<float>(arch, new IconVSROptions());
model.Train(lowResClip, highResClip); // [15, 3, 64, 64] -> [15, 3, 256, 256], pixel values 0-255
var upscaled = model.Upscale(lowResClip);
Constructors
| Constructor | Summary |
|---|---|
IconVSR(NeuralNetworkArchitecture<T> architecture, IconVSROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates an IconVSR model in native training mode. |
IconVSR(NeuralNetworkArchitecture<T> architecture, string modelPath, IconVSROptions options) |
Creates an IconVSR model in native training mode. |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
Frames are pixels on the 0-255 scale; the network divides them by 255. |
LayerInputDomain GetOutputDomain(int[] outputShape) |
Upscaled frames come back on the same 0-255 scale. |
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Tensor<T> Upscale(Tensor<T> lowResFrames) |