AdaSpeech2<T>
Models & Types · AiDotNet.TextToSpeech.Classic
AdaSpeech 2: adaptive TTS that adapts to a new voice from untranscribed speech, through a mel-spectrogram encoder aligned to the phoneme encoder’s output space.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VarianceAdaptorTtsModelBase<T> → AdaSpeech<T> → AdaSpeech2<T>
For Beginners
AdaSpeech 2 can learn a new voice from recordings that have no transcript: it learns to read a spectrogram into the same internal representation it reads text into, then practises reproducing the new speaker’s recordings.
How It Works
References: “AdaSpeech 2: Adaptive Text to Speech with Untranscribed Data” (Yan et al., ICASSP 2021).
The TTS pipeline is AdaSpeech’s (§2.1, AdaSpeech): phoneme encoder, acoustic condition modeling,
variance adaptor and a mel decoder with conditional layer normalization. AdaSpeech 2 adds a mel-spectrogram encoder
of 4 feed-forward Transformer blocks (“considering the symmetry of the system”) and a four-step pipeline (§2, Fig. 2),
selected with CurrentStep:
The paper does not say how the mel encoder reads 80-bin frames into its 256-wide blocks; a linear projection and the sinusoidal positions of the phoneme encoder’s input do it. In reconstruction the speaker embedding, the utterance-level vector and the frame pitch and energy embeddings are added as in synthesis; the phoneme-level vectors are not, since untranscribed speech has no phoneme alignment to average frames over.
Constructors
| Constructor | Summary |
|---|---|
AdaSpeech2(NeuralNetworkArchitecture<T> architecture, AdaSpeech2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a trainable AdaSpeech 2. |
AdaSpeech2(NeuralNetworkArchitecture<T> architecture, string modelPath, AdaSpeech2Options options) |
Creates a trainable AdaSpeech 2. |
Properties
| Property | Summary |
|---|---|
AdaSpeech2TrainingStep CurrentStep { get; set; } |
The step of the adaptation pipeline (§2, Fig. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
T EvaluateUntranscribed(Tensor<T> mel, int speakerId, Tensor<T> pitch, Tensor<T> energy) |
The untranscribed reconstruction loss on mel, without updating the model. |
T TrainUntranscribed(Tensor<T> audio, int speakerId) |
One untranscribed-speech adaptation step on a recording, deriving its mel spectrogram, WORLD pitch and frame energy as the source model’s training does. |
T TrainUntranscribed(Tensor<T> mel, int speakerId, Tensor<T> pitch, Tensor<T> energy) |
One untranscribed-speech adaptation step (§2.3): reconstructs mel through the mel encoder and the mel decoder and updates only the conditional layer normalizations and the speaker embedding. |