FreGrad<T>
Models & Types · AiDotNet.TextToSpeech.Vocoders
FreGrad: a lightweight and fast frequency-aware diffusion vocoder that denoises the waveform’s two Haar wavelet sub-bands with frequency-aware dilated convolutions.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VocoderBase<T> → SegmentVocoderBase<T> → DiffusionVocoderBase<T> → FreGrad<T>
For Beginners
FreGrad splits audio into a low-pitched and a high-pitched half-length signal, which are simpler to clean up than the full waveform, removes noise from both, and recombines them losslessly.
How It Works
References: “FreGrad: Lightweight and Fast Frequency-aware Diffusion Vocoder” (Nguyen et al., ICASSP 2024) and kaistmm/fregrad for what the paper leaves unstated.
The waveform is split by the Haar DWT into low and high sub-bands of half its length (§3.1); each band is diffused with noise from its own PriorGrad-style energy prior, from the lower or the upper half of the mel bands (§3.3); a DiffWave network whose dilated convolutions are Freq-DConvs (§3.2), taking and predicting both bands, estimates the noise; the loss is Σ_{l,h} (‖ε − ε̂‖²_{Σ⁻¹} + λ L_mag(ε, ε̂)) with L_mag the multi-resolution STFT log-magnitude loss (Eq. 9–10); the β schedule is shifted to zero terminal SNR (Eq. 8). Sampling denoises both bands from their priors and returns their inverse DWT.
Constructors
| Constructor | Summary |
|---|---|
FreGrad(NeuralNetworkArchitecture<T> architecture, FreGradOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a trainable FreGrad. |
FreGrad(NeuralNetworkArchitecture<T> architecture, string modelPath, FreGradOptions options) |
Creates a trainable FreGrad. |
Properties
| Property | Summary |
|---|---|
int UpsampleFactor { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> PriorStd(Tensor<T> mel) |
The priors’ standard deviations [1, 2, frames · hop / 2] for a mel spectrogram [1, mel, frames]: the low band’s from the lower half of the mel bands, the high band’s from the upper half, each normalized as PriorGrad’s and repeated over… |
void FitEnergyStatistics(IEnumerable<Tensor<T>> recordings, bool useDataMaximum) |
Sets the energy statistics of the priors from training recordings (see FitEnergyStatistics). |