FastSpeech2<T>
Models & Types · AiDotNet.TextToSpeech.Classic
FastSpeech 2: non-autoregressive TTS with a variance adaptor for duration, pitch and energy.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VarianceAdaptorTtsModelBase<T> → FastSpeech2<T>
Implements: IAcousticModel<T>
For Beginners
FastSpeech 2 reads phonemes, decides how long each one lasts and how high and loud it is, stretches the phoneme features to that many frames, and turns them into a mel spectrogram all at once. A vocoder (e.g. HiFi-GAN) turns the spectrogram into audio.
How It Works
References: “FastSpeech 2: Fast and High-Quality End-to-End Text to Speech” (Ren et al., 2021).
The layer stack is the paper’s (§2.2, App. A): phoneme embedding and sinusoidal positions, 4 feed-forward
Transformer blocks (hidden 256, 2 heads, 1D convolutions of kernel 9 and 1 with 1024 filters), the variance
adaptor (duration, pitch as a CWT spectrogram, energy; VarianceAdaptorLayer), positions again,
4 more FFT blocks and a linear projection to 80 mel channels. The model is trained on the sum of the mel MAE and
the MSE of each variance predictor, with ground-truth duration, pitch and energy driving the adaptor (§2.3).
Training data: durations come from an external forced aligner (MFA in the paper), so a plain
Train(tokens, mel) cannot train this model faithfully and throws; pass a TtsTrainingSample
with Durations. Pitch (WORLD DIO + StoneMask), energy (STFT frame norm)
and the mel target are derived from Audio when not supplied, as the paper
derives them.
Constructors
| Constructor | Summary |
|---|---|
FastSpeech2(NeuralNetworkArchitecture<T> architecture, FastSpeech2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
FastSpeech2(NeuralNetworkArchitecture<T> architecture, string modelPath, FastSpeech2Options options) |
Properties
| Property | Summary |
|---|---|
int FftSize { get; } |
|
int HopSize { get; } |
|
int MaxTextLength { get; } |
|
int MelChannels { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> TextToMel(string text) |
Generates a mel spectrogram from text (FastSpeech 2 is an acoustic model; pair it with a vocoder). |
void LoadConvertedWeights(string path) |
Copies {ConvertedParameterTensor} from source straight into a parameter vector. |