FloRNN<T>

Models & Types · AiDotNet.Video.Denoising

FloRNN: unidirectional video denoising with a look-ahead recurrent module that mimics a backward one (Li et al., ECCV 2022).

Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoDenoisingBase<T> → FloRNN<T>

For Beginners

FloRNN denoises video frame by frame, but it also looks a few frames into the future and warps what it saw there back to the current frame, getting most of the benefit of a two-way model while still running in a single pass.

How It Works

The network follows the authors’ release (github.com/nagejacob/FloRNN, models/flornn.py). Two recurrent modules, each a 3x3 convolution followed by residual blocks (conv-ReLU-conv plus identity, no biases), read the noisy frame, its noise-level map and the previous hidden state:

The sequence is reflected past its end so the last frames have frames to look ahead to. The paper’s flow is a pretrained PWC-Net that is never trained; here a SPyNet estimator plays that role and is likewise excluded from training (load pretrained weights into it for meaningful alignment). Weights are Kaiming-normal scaled by 0.2, as in the release. Training feeds the noise level the noise was drawn with (the RMS of noisy minus clean) and scores frames 1 to T-ForwardCount-1, as the authors’ script does; their summed MSE over 2N is a constant multiple of the mean used here.

Example

var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.ThreeDimensional,
    taskType: NeuralNetworkTaskType.Generative,
    inputHeight: 96, inputWidth: 96, inputDepth: 3);
var flornn = new FloRNN<double>(architecture, new FloRNNOptions());
var clean = flornn.Denoise(noisyFrames, noiseLevel: 25.0 / 255.0); // [T, C, H, W] in [0, 1]

Constructors

Constructor Summary
FloRNN(NeuralNetworkArchitecture<T> architecture, FloRNNOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a FloRNN model for native training and inference.
FloRNN(NeuralNetworkArchitecture<T> architecture, string modelPath, FloRNNOptions options) Creates a FloRNN model for native training and inference.

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape) The released FloRNN model consumes and emits normalized image tensors.
LayerInputDomain GetOutputDomain(int[] outputShape)
ModelMetadata<T> GetModelMetadata()
Tensor<T> Denoise(Tensor<T> noisyFrames)
Tensor<T> Denoise(Tensor<T> noisyFrames, double noiseLevel) Denoises frames [T, C, H, W] (or [N, T, C, H, W]) corrupted by noise of the given standard deviation, the value the noise-level map carries.
Tensor<T> ForwardForTraining(Tensor<T> input)