ProDiff<T>
Models & Types · AiDotNet.TextToSpeech.Classic
ProDiff: a progressive fast diffusion model that synthesizes a mel spectrogram in two denoising steps by predicting the clean spectrogram directly and distilling a 4-step teacher.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VarianceAdaptorTtsModelBase<T> → ProDiff<T>
Implements: IAcousticModel<T>
For Beginners
Diffusion models usually clean up a noisy spectrogram over hundreds of small steps. ProDiff predicts the clean spectrogram outright and learns from a slower teacher, so two steps suffice.
How It Works
References: “ProDiff: Progressive Fast Diffusion Model for High-Quality Text-to-Speech” (Huang et al., ACM MM 2022) and its reference implementation (Rongjiehuang/ProDiff) for what the paper leaves unstated.
The phoneme encoder and variance adaptor are FastSpeech 2’s (§4.4): a pre-net, FFT blocks and a linear projection,
then duration, pitch-spectrogram and energy predictors with a length regulator. The spectrogram denoiser
f_θ(x_t | t, c) is a non-causal WaveNet conditioned on the adaptor output (ProDiffDenoiser). Diffusion uses a
cosine schedule with α_t = √ᾱ_t and σ_t = √(1 − ᾱ_t). The teacher (Teacher)
minimizes ‖f_θ(α_t x₀ + σ_t ε) − x₀‖² plus 1 − SSIM (Eq. 9, 11) with T₁ = 4 steps; the distillation phase
(Algorithm 1) samples t ∈ S·{1..T₂}, S = T₁/T₂, runs two DDIM steps of the frozen teacher from x_t to x_{t−S},
solves for the target x̂₀ that one student step would need, and minimizes ‖f_θ(x_t) − x̂₀‖² + 1 − SSIM (Eq. 10, 11).
Both add the variance reconstruction losses with weight 0.1 (Eq. 12). Sampling (Algorithm 2) starts from N(0, I)
and alternates x̂₀ = f_θ(x_t) with a draw from the posterior q(x_{t−S} | x_t, x̂₀).
Constructors
| Constructor | Summary |
|---|---|
ProDiff(NeuralNetworkArchitecture<T> architecture, ProDiffOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
ProDiff(NeuralNetworkArchitecture<T> architecture, string modelPath, ProDiffOptions options) |
Properties
| Property | Summary |
|---|---|
ProDiffTrainingPhase CurrentPhase { get; set; } |
The training stage: Teacher until the teacher converges, then BeginDistillation. |
int FftSize { get; } |
|
int HopSize { get; } |
|
int MaxTextLength { get; } |
|
int MelChannels { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> TextToMel(string text) |
Generates a mel spectrogram from text (the paper pairs it with HiFi-GAN). |
void BeginDistillation() |
Starts distillation from this model’s own trained teacher: the student keeps the current parameters (“we first initialize ProDiff with a copy of the teacher”, §4.3) and a frozen copy of the denoiser becomes the teacher. |
void BeginDistillation(ProDiff<T> teacher) |
Starts distillation from a separately trained teacher: this model is initialized with a copy of the teacher’s parameters (§4.3) and the teacher’s denoiser is frozen as the distillation target. |