ProDiff<T>

Models & Types · AiDotNet.TextToSpeech.Classic

ProDiff: a progressive fast diffusion model that synthesizes a mel spectrogram in two denoising steps by predicting the clean spectrogram directly and distilling a 4-step teacher.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VarianceAdaptorTtsModelBase<T> → ProDiff<T>

Implements: IAcousticModel<T>

For Beginners

Diffusion models usually clean up a noisy spectrogram over hundreds of small steps. ProDiff predicts the clean spectrogram outright and learns from a slower teacher, so two steps suffice.

How It Works

References: “ProDiff: Progressive Fast Diffusion Model for High-Quality Text-to-Speech” (Huang et al., ACM MM 2022) and its reference implementation (Rongjiehuang/ProDiff) for what the paper leaves unstated.

The phoneme encoder and variance adaptor are FastSpeech 2’s (§4.4): a pre-net, FFT blocks and a linear projection, then duration, pitch-spectrogram and energy predictors with a length regulator. The spectrogram denoiser f_θ(x_t | t, c) is a non-causal WaveNet conditioned on the adaptor output (ProDiffDenoiser). Diffusion uses a cosine schedule with α_t = √ᾱ_t and σ_t = √(1 − ᾱ_t). The teacher (Teacher) minimizes ‖f_θ(α_t x₀ + σ_t ε) − x₀‖² plus 1 − SSIM (Eq. 9, 11) with T₁ = 4 steps; the distillation phase (Algorithm 1) samples t ∈ S·{1..T₂}, S = T₁/T₂, runs two DDIM steps of the frozen teacher from x_t to x_{t−S}, solves for the target x̂₀ that one student step would need, and minimizes ‖f_θ(x_t) − x̂₀‖² + 1 − SSIM (Eq. 10, 11). Both add the variance reconstruction losses with weight 0.1 (Eq. 12). Sampling (Algorithm 2) starts from N(0, I) and alternates x̂₀ = f_θ(x_t) with a draw from the posterior q(x_{t−S} | x_t, x̂₀).

Constructors

Constructor Summary
ProDiff(NeuralNetworkArchitecture<T> architecture, ProDiffOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
ProDiff(NeuralNetworkArchitecture<T> architecture, string modelPath, ProDiffOptions options)

Properties

Property Summary
ProDiffTrainingPhase CurrentPhase { get; set; } The training stage: Teacher until the teacher converges, then BeginDistillation.
int FftSize { get; }
int HopSize { get; }
int MaxTextLength { get; }
int MelChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> TextToMel(string text) Generates a mel spectrogram from text (the paper pairs it with HiFi-GAN).
void BeginDistillation() Starts distillation from this model’s own trained teacher: the student keeps the current parameters (“we first initialize ProDiff with a copy of the teacher”, §4.3) and a frozen copy of the denoiser becomes the teacher.
void BeginDistillation(ProDiff<T> teacher) Starts distillation from a separately trained teacher: this model is initialized with a copy of the teacher’s parameters (§4.3) and the teacher’s denoiser is frozen as the distillation target.