AdaSpeech<T>
Models & Types · AiDotNet.TextToSpeech.Classic
AdaSpeech: adaptive TTS for custom voice with acoustic condition modeling and conditional layer normalization.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VarianceAdaptorTtsModelBase<T> → AdaSpeech<T>
Implements: IAcousticModel<T>
Known derived types: AdaSpeech2<T>
For Beginners
AdaSpeech is a FastSpeech 2 that can learn a new voice from a few minutes of speech by adjusting only a few thousand numbers per speaker.
How It Works
References: “AdaSpeech: Adaptive Text to Speech for Custom Voice” (Chen et al., ICLR 2021).
The backbone is FastSpeech 2 (§2): a phoneme encoder of 4 FFT blocks, the variance adaptor (duration, pitch, energy), and a mel decoder of 4 FFT blocks, hidden 256. Two additions make it adaptable:
Training follows §3 through CurrentPhase: source pre-training (everything but the phoneme-level
predictor), joint training with the predictor’s MSE to the stop-gradient phoneme-level encoder output, and
adaptation to a new voice, which updates only the speaker embedding and the conditional layer normalizations’
W_γ, W_β. The loss is FastSpeech 2’s: mel MAE plus the variance predictors’ MSE.
Training data: durations from a forced aligner (MFA in the paper) and the speaker’s index, so a plain
Train(tokens, mel) throws; pass a TtsTrainingSample with
Durations and SpeakerId. Synthesis reads
Voice: the speaker and a reference mel spectrogram of them.
Constructors
| Constructor | Summary |
|---|---|
AdaSpeech(NeuralNetworkArchitecture<T> architecture, AdaSpeechOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a trainable AdaSpeech. |
AdaSpeech(NeuralNetworkArchitecture<T> architecture, string modelPath, AdaSpeechOptions options) |
Creates a trainable AdaSpeech. |
Properties
| Property | Summary |
|---|---|
AdaSpeechTrainingPhase CurrentPhase { get; set; } |
The training phase (§3): source-model pre-training without the phoneme-level predictor, then joint training with it, then per-voice adaptation of the conditional layer normalizations and the speaker embedding. |
int FftSize { get; } |
|
int HopSize { get; } |
|
int MaxTextLength { get; } |
|
int MelChannels { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> TextToMel(string text) |
Generates a mel spectrogram from text in Voice (pair it with a vocoder; the paper uses MelGAN). |
void LoadConvertedWeights(string path) |
Copies {ConvertedParameterTensor} from source straight into a parameter vector. |