FreGrad<T>

Models & Types · AiDotNet.TextToSpeech.Vocoders

FreGrad: a lightweight and fast frequency-aware diffusion vocoder that denoises the waveform’s two Haar wavelet sub-bands with frequency-aware dilated convolutions.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VocoderBase<T> → SegmentVocoderBase<T> → DiffusionVocoderBase<T> → FreGrad<T>

For Beginners

FreGrad splits audio into a low-pitched and a high-pitched half-length signal, which are simpler to clean up than the full waveform, removes noise from both, and recombines them losslessly.

How It Works

References: “FreGrad: Lightweight and Fast Frequency-aware Diffusion Vocoder” (Nguyen et al., ICASSP 2024) and kaistmm/fregrad for what the paper leaves unstated.

The waveform is split by the Haar DWT into low and high sub-bands of half its length (§3.1); each band is diffused with noise from its own PriorGrad-style energy prior, from the lower or the upper half of the mel bands (§3.3); a DiffWave network whose dilated convolutions are Freq-DConvs (§3.2), taking and predicting both bands, estimates the noise; the loss is Σ_{l,h} (‖ε − ε̂‖²_{Σ⁻¹} + λ L_mag(ε, ε̂)) with L_mag the multi-resolution STFT log-magnitude loss (Eq. 9–10); the β schedule is shifted to zero terminal SNR (Eq. 8). Sampling denoises both bands from their priors and returns their inverse DWT.

Constructors

Constructor Summary
FreGrad(NeuralNetworkArchitecture<T> architecture, FreGradOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a trainable FreGrad.
FreGrad(NeuralNetworkArchitecture<T> architecture, string modelPath, FreGradOptions options) Creates a trainable FreGrad.

Properties

Property Summary
int UpsampleFactor { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> PriorStd(Tensor<T> mel) The priors’ standard deviations [1, 2, frames · hop / 2] for a mel spectrogram [1, mel, frames]: the low band’s from the lower half of the mel bands, the high band’s from the upper half, each normalized as PriorGrad’s and repeated over…
void FitEnergyStatistics(IEnumerable<Tensor<T>> recordings, bool useDataMaximum) Sets the energy statistics of the priors from training recordings (see FitEnergyStatistics).