DiffusionVocoderBase<T>

Base Classes · AiDotNet.TextToSpeech.Vocoders

The denoising-diffusion training and sampling diffusion vocoders share (Ho et al. 2020; DiffWave, WaveGrad, PriorGrad, FreGrad): a network ε_θ predicts the noise added to a waveform at a noise level, conditioned on the mel spectrogram, and the reverse process turns noise into speech.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VocoderBase<T> → SegmentVocoderBase<T> → DiffusionVocoderBase<T>

Known derived types: DiffWave<T>, FreGrad<T>, PriorGrad<T>, WaveGrad<T>

For Beginners

The model learns to remove a little noise at a time; to make speech it starts from pure noise and removes noise step by step, guided by the spectrogram.

How It Works

Training (Algorithm 1 of DDPM and the vocoder papers): crop a segment, draw a noise level, form x_t = √ᾱ_t x₀ + √(1 − ᾱ_t) ε and minimize the model’s loss between ε and ε_θ(x_t, level, mel). Sampling (Algorithm 2): from x_N ~ the prior, x_{n−1} = (x_n − β_n / √(1 − ᾱ_n) · ε_θ) / √α_n + σ_n z with σ_n = √((1 − ᾱ_{n−1}) / (1 − ᾱ_n) · β_n), over the training schedule or a shorter inference schedule.

A model supplies its network (Denoise), its input features, its schedules and how the noise level conditions the network, and may change the noise prior and the loss (PriorGrad’s data-dependent prior and Mahalanobis loss) and the space the diffusion runs in (FreGrad’s wavelet sub-bands).