AdaSpeech<T>

Models & Types · AiDotNet.TextToSpeech.Classic

AdaSpeech: adaptive TTS for custom voice with acoustic condition modeling and conditional layer normalization.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VarianceAdaptorTtsModelBase<T> → AdaSpeech<T>

Implements: IAcousticModel<T>

Known derived types: AdaSpeech2<T>

For Beginners

AdaSpeech is a FastSpeech 2 that can learn a new voice from a few minutes of speech by adjusting only a few thousand numbers per speaker.

How It Works

References: “AdaSpeech: Adaptive Text to Speech for Custom Voice” (Chen et al., ICLR 2021).

The backbone is FastSpeech 2 (§2): a phoneme encoder of 4 FFT blocks, the variance adaptor (duration, pitch, energy), and a mel decoder of 4 FFT blocks, hidden 256. Two additions make it adaptable:

Training follows §3 through CurrentPhase: source pre-training (everything but the phoneme-level predictor), joint training with the predictor’s MSE to the stop-gradient phoneme-level encoder output, and adaptation to a new voice, which updates only the speaker embedding and the conditional layer normalizations’ W_γ, W_β. The loss is FastSpeech 2’s: mel MAE plus the variance predictors’ MSE.

Training data: durations from a forced aligner (MFA in the paper) and the speaker’s index, so a plain Train(tokens, mel) throws; pass a TtsTrainingSample with Durations and SpeakerId. Synthesis reads Voice: the speaker and a reference mel spectrogram of them.

Constructors

Constructor Summary
AdaSpeech(NeuralNetworkArchitecture<T> architecture, AdaSpeechOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a trainable AdaSpeech.
AdaSpeech(NeuralNetworkArchitecture<T> architecture, string modelPath, AdaSpeechOptions options) Creates a trainable AdaSpeech.

Properties

Property Summary
AdaSpeechTrainingPhase CurrentPhase { get; set; } The training phase (§3): source-model pre-training without the phoneme-level predictor, then joint training with it, then per-voice adaptation of the conditional layer normalizations and the speaker embedding.
int FftSize { get; }
int HopSize { get; }
int MaxTextLength { get; }
int MelChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> TextToMel(string text) Generates a mel spectrogram from text in Voice (pair it with a vocoder; the paper uses MelGAN).
void LoadConvertedWeights(string path) Copies {ConvertedParameterTensor} from source straight into a parameter vector.