ShiftNet<T>

Models & Types · AiDotNet.Video.Denoising

Shift-Net: video denoising with grouped spatial-temporal shift (Li et al., CVPR 2023).

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoDenoisingBase<T> → ShiftNet<T>

For Beginners

Instead of estimating motion, Shift-Net slides copies of each frame’s features to the neighbouring frames and by a few pixels in many directions, and lets the following layers pick the slide that lines up. Sliding costs nothing, so the model is fast.

How It Works

The network is the authors’ denoising release (github.com/dasongli1/Shift-Net, gshift_denoise1.py, GShiftNet). The frames of a clip are processed as a batch:

Training follows options/davis_denoise.yml: BasicSR’s PSNR loss (weight 0.5), Adam at 4e-4 with betas (0.9, 0.99) cosine-annealed to 1e-7 per iteration. The noise map carries the RMS of noisy minus clean, the σ the noise was drawn with. Frames must be a multiple of 8 in each spatial size or are zero-padded to one.

The paper’s text describes a displacement set of {-9, -5, 0, 5, 9}; the released network, whose numbers the paper reports, shifts by multiples of s = 4, and that is what is built here (see GroupedSpatialTemporalShift for the text’s version).

Example

var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.ThreeDimensional,
    taskType: NeuralNetworkTaskType.Generative,
    inputHeight: 256, inputWidth: 256, inputDepth: 3);
var shiftNet = new ShiftNet<double>(architecture, new ShiftNetOptions());
var clean = shiftNet.Denoise(noisyFrames); // [T, 3, H, W], pixel values 0-255

Constructors

Constructor Summary
ShiftNet(NeuralNetworkArchitecture<T> architecture, ShiftNetOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a ShiftNet model for native training and inference.
ShiftNet(NeuralNetworkArchitecture<T> architecture, string modelPath, ShiftNetOptions options) Creates a ShiftNet model for native training and inference.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> Denoise(Tensor<T> noisyFrames)
Tensor<T> ForwardForTraining(Tensor<T> input)