FastSpeech2<T>

Models & Types · AiDotNet.TextToSpeech.Classic

FastSpeech 2: non-autoregressive TTS with a variance adaptor for duration, pitch and energy.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VarianceAdaptorTtsModelBase<T> → FastSpeech2<T>

Implements: IAcousticModel<T>

For Beginners

FastSpeech 2 reads phonemes, decides how long each one lasts and how high and loud it is, stretches the phoneme features to that many frames, and turns them into a mel spectrogram all at once. A vocoder (e.g. HiFi-GAN) turns the spectrogram into audio.

How It Works

References: “FastSpeech 2: Fast and High-Quality End-to-End Text to Speech” (Ren et al., 2021).

The layer stack is the paper’s (§2.2, App. A): phoneme embedding and sinusoidal positions, 4 feed-forward Transformer blocks (hidden 256, 2 heads, 1D convolutions of kernel 9 and 1 with 1024 filters), the variance adaptor (duration, pitch as a CWT spectrogram, energy; VarianceAdaptorLayer), positions again, 4 more FFT blocks and a linear projection to 80 mel channels. The model is trained on the sum of the mel MAE and the MSE of each variance predictor, with ground-truth duration, pitch and energy driving the adaptor (§2.3).

Training data: durations come from an external forced aligner (MFA in the paper), so a plain Train(tokens, mel) cannot train this model faithfully and throws; pass a TtsTrainingSample with Durations. Pitch (WORLD DIO + StoneMask), energy (STFT frame norm) and the mel target are derived from Audio when not supplied, as the paper derives them.

Constructors

Constructor Summary
FastSpeech2(NeuralNetworkArchitecture<T> architecture, FastSpeech2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
FastSpeech2(NeuralNetworkArchitecture<T> architecture, string modelPath, FastSpeech2Options options)

Properties

Property Summary
int FftSize { get; }
int HopSize { get; }
int MaxTextLength { get; }
int MelChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> TextToMel(string text) Generates a mel spectrogram from text (FastSpeech 2 is an acoustic model; pair it with a vocoder).
void LoadConvertedWeights(string path) Copies {ConvertedParameterTensor} from source straight into a parameter vector.