UDVD<T>
Models & Types · AiDotNet.Video.Denoising
UDVD: unsupervised deep video denoising with a blind-spot network (Sheth et al., ICCV 2021).
Inheritance: NeuralNetworkBase<T> → VideoNeuralNetworkBase<T> → VideoDenoisingBase<T> → UDVD<T>
For Beginners
UDVD learns from noisy video alone. Its network can never see the pixel it is predicting, so the best it can do is predict the clean signal from the neighbourhood and nearby frames, which is exactly what denoising needs; the noisy pixel is then folded back in with Bayes’ rule.
How It Works
The network is the authors’ “blind-video-net-4” (github.com/sreyas-mohan/udvd, models/udvd.py). Five contiguous frames predict the middle one. Every convolution is blind: the input is shifted down one row before a replicate-padded 3x3 convolution and the extra row cropped after, so each output sees only pixels above it, and max-pooling is preceded by the same shift. The frames are stacked in four rotations, so the four copies together see everything except the centre pixel; a final shift removes it. A first blind U-Net maps each of the frame triplets (1-3, 2-4, 3-5) to features, a second maps their concatenation, the rotations are undone and stacked on the channels, and three 1x1 convolutions produce, per pixel, the mean μ and a factor A of the covariance Σ = AᵀA of the clean value.
Training maximizes the Gaussian log-likelihood of the target frame under N(μ, AᵀA + σ²I) (utils/loss_funtions.py, “loglike”), with the release’s -0.1σ regularizer. Passing the noisy frames as the target is the paper’s unsupervised training; a clean target makes it supervised. The denoised frame is the posterior mean E[x|y] = (Σ⁻¹ + σ⁻²I)⁻¹(Σ⁻¹μ + σ⁻²y) (utils/post_processing.py). Adam at 1e-4 halved at epochs 20, 25 and 30.
Example
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.ThreeDimensional,
taskType: NeuralNetworkTaskType.Generative,
inputHeight: 128, inputWidth: 128, inputDepth: 3);
var udvd = new UDVD<double>(architecture, new UDVDOptions());
udvd.Train(noisyFrames, noisyFrames); // unsupervised: the noisy video is its own target
var clean = udvd.Denoise(noisyFrames); // [T, 3, H, W], pixel values 0-255
Constructors
| Constructor | Summary |
|---|---|
UDVD(NeuralNetworkArchitecture<T> architecture, UDVDOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a UDVD model for native training and inference. |
UDVD(NeuralNetworkArchitecture<T> architecture, string modelPath, UDVDOptions options) |
Creates a UDVD model for native training and inference. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> Denoise(Tensor<T> noisyFrames) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |