PriorGrad<T>

Models & Types · AiDotNet.TextToSpeech.Vocoders

PriorGrad vocoder: DiffWave whose diffusion prior is a data-dependent diagonal Gaussian N(0, Σ_c) with the standard deviation of each frame taken from the mel spectrogram’s normalized frame energy.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VocoderBase<T> → SegmentVocoderBase<T> → DiffusionVocoderBase<T> → PriorGrad<T>

For Beginners

DiffWave starts every synthesis from the same plain noise; PriorGrad starts from noise that is already loud where the speech is loud and quiet where it is quiet, so there is less left to learn.

How It Works

References: “PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Dependent Adaptive Prior” (Lee et al., ICLR 2022) and microsoft/NeuralSpeech PriorGrad-vocoder for what the paper leaves unstated.

Training (Algorithm 1): ε ~ N(0, Σ), t uniform, x_t = √ᾱ_t x₀ + √(1 − ᾱ_t) ε and the Mahalanobis loss ‖ε − ε_θ(x_t, c, t)‖²_{Σ⁻¹}. Sampling (Algorithm 2): x_T ~ N(0, Σ) and every step’s noise z ~ N(0, Σ). Σ is the frame energy √Σ_bands exp(mel) normalized to (0, 1], clipped below at 0.1 and repeated over each frame’s hop (§4). The network is DiffWave’s, unchanged.

Constructors

Constructor Summary
PriorGrad(NeuralNetworkArchitecture<T> architecture, PriorGradOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a trainable PriorGrad.
PriorGrad(NeuralNetworkArchitecture<T> architecture, string modelPath, PriorGradOptions options) Creates a trainable PriorGrad.

Properties

Property Summary
int UpsampleFactor { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> PriorStd(Tensor<T> mel) The prior’s standard deviation per sample [1, 1, samples] for a mel spectrogram [1, mel, frames]: each frame’s energy normalized to (0, 1], clipped below at the minimum standard deviation, repeated over the hop.
void FitEnergyStatistics(IEnumerable<Tensor<T>> recordings, bool useDataMaximum) Sets EnergyMin (and, with useDataMaximum, EnergyMax) to the extremes of the recordings’ frame energies, as the reference computes them over the training set.