ShiftNet<T>
Models & Types · AiDotNet.Video.Denoising
Shift-Net: video denoising with grouped spatial-temporal shift (Li et al., CVPR 2023).
Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoDenoisingBase<T> → ShiftNet<T>
For Beginners
Instead of estimating motion, Shift-Net slides copies of each frame’s features to the neighbouring frames and by a few pixels in many directions, and lets the following layers pick the slide that lines up. Sliding costs nothing, so the model is fast.
How It Works
The network is the authors’ denoising release (github.com/dasongli1/Shift-Net, gshift_denoise1.py, GShiftNet). The frames of a clip are processed as a batch:
Training follows options/davis_denoise.yml: BasicSR’s PSNR loss (weight 0.5), Adam at 4e-4 with betas (0.9, 0.99) cosine-annealed to 1e-7 per iteration. The noise map carries the RMS of noisy minus clean, the σ the noise was drawn with. Frames must be a multiple of 8 in each spatial size or are zero-padded to one.
The paper’s text describes a displacement set of {-9, -5, 0, 5, 9}; the released network, whose numbers the paper
reports, shifts by multiples of s = 4, and that is what is built here (see
GroupedSpatialTemporalShift for the text’s version).
Example
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.ThreeDimensional,
taskType: NeuralNetworkTaskType.Generative,
inputHeight: 256, inputWidth: 256, inputDepth: 3);
var shiftNet = new ShiftNet<double>(architecture, new ShiftNetOptions());
var clean = shiftNet.Denoise(noisyFrames); // [T, 3, H, W], pixel values 0-255
Constructors
| Constructor | Summary |
|---|---|
ShiftNet(NeuralNetworkArchitecture<T> architecture, ShiftNetOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a ShiftNet model for native training and inference. |
ShiftNet(NeuralNetworkArchitecture<T> architecture, string modelPath, ShiftNetOptions options) |
Creates a ShiftNet model for native training and inference. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> Denoise(Tensor<T> noisyFrames) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |