GradTTS<T>

Models & Types · AiDotNet.TextToSpeech.Classic

Grad-TTS: text-to-speech with a score-based diffusion decoder that turns an aligned Gaussian prior into a mel spectrogram.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → GradTTS<T>

Implements: IAcousticModel<T>

For Beginners

Grad-TTS first sketches a blurry spectrogram from the text, then refines it step by step by removing noise, the way image diffusion models paint pictures.

How It Works

References: “Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech” (Popov et al., ICML 2021) and its reference implementation (huawei-noah/Speech-Backbones) for what the paper leaves unstated.

The encoder is Glow-TTS’s (§3): it predicts per-token prior means μ and durations, aligned to the mel spectrogram by monotonic alignment search during training. The decoder is a score network s_θ(X_t, μ, t) (GradTtsScoreEstimator, a U-Net) for the forward diffusion dX = ½ β_t (μ − X) dt + √β_t dW with β_t = 0.05 + 19.95 t, whose terminal distribution is N(μ, I). Training minimizes the duration loss, the prior loss ½‖y − μ‖² and the score-matching loss on random 2-second segments (MAS and durations use the whole utterance), with encoder and decoder gradients each clipped to norm 1. Inference draws X₁ = μ + ε/τ (τ = 1.5) and solves the reverse ODE with N Euler steps.

The model aligns itself, so a plain Train(tokens, mel) is its full training signal.

Constructors

Constructor Summary
GradTTS(NeuralNetworkArchitecture<T> architecture, GradTTSOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
GradTTS(NeuralNetworkArchitecture<T> architecture, string modelPath, GradTTSOptions options)

Properties

Property Summary
int FftSize { get; }
int HopSize { get; }
int MaxTextLength { get; }
int MelChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> TextToMel(string text) Generates a mel spectrogram from text (the paper pairs it with HiFi-GAN).