AdaSpeech2<T>

Models & Types · AiDotNet.TextToSpeech.Classic

AdaSpeech 2: adaptive TTS that adapts to a new voice from untranscribed speech, through a mel-spectrogram encoder aligned to the phoneme encoder’s output space.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VarianceAdaptorTtsModelBase<T> → AdaSpeech<T> → AdaSpeech2<T>

For Beginners

AdaSpeech 2 can learn a new voice from recordings that have no transcript: it learns to read a spectrogram into the same internal representation it reads text into, then practises reproducing the new speaker’s recordings.

How It Works

References: “AdaSpeech 2: Adaptive Text to Speech with Untranscribed Data” (Yan et al., ICASSP 2021).

The TTS pipeline is AdaSpeech’s (§2.1, AdaSpeech): phoneme encoder, acoustic condition modeling, variance adaptor and a mel decoder with conditional layer normalization. AdaSpeech 2 adds a mel-spectrogram encoder of 4 feed-forward Transformer blocks (“considering the symmetry of the system”) and a four-step pipeline (§2, Fig. 2), selected with CurrentStep:

The paper does not say how the mel encoder reads 80-bin frames into its 256-wide blocks; a linear projection and the sinusoidal positions of the phoneme encoder’s input do it. In reconstruction the speaker embedding, the utterance-level vector and the frame pitch and energy embeddings are added as in synthesis; the phoneme-level vectors are not, since untranscribed speech has no phoneme alignment to average frames over.

Constructors

Constructor Summary
AdaSpeech2(NeuralNetworkArchitecture<T> architecture, AdaSpeech2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a trainable AdaSpeech 2.
AdaSpeech2(NeuralNetworkArchitecture<T> architecture, string modelPath, AdaSpeech2Options options) Creates a trainable AdaSpeech 2.

Properties

Property Summary
AdaSpeech2TrainingStep CurrentStep { get; set; } The step of the adaptation pipeline (§2, Fig.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
T EvaluateUntranscribed(Tensor<T> mel, int speakerId, Tensor<T> pitch, Tensor<T> energy) The untranscribed reconstruction loss on mel, without updating the model.
T TrainUntranscribed(Tensor<T> audio, int speakerId) One untranscribed-speech adaptation step on a recording, deriving its mel spectrogram, WORLD pitch and frame energy as the source model’s training does.
T TrainUntranscribed(Tensor<T> mel, int speakerId, Tensor<T> pitch, Tensor<T> energy) One untranscribed-speech adaptation step (§2.3): reconstructs mel through the mel encoder and the mel decoder and updates only the conditional layer normalizations and the speaker embedding.