PromptTTS<T>

Models & Types · AiDotNet.TextToSpeech.DescriptionBased

PromptTTS: speech whose style a plain-language prompt describes (Guo et al. 2023).

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VarianceAdaptorTtsModelBase<T> → PromptTTS<T>

Implements: IAcousticModel<T>

For Beginners

You describe how the speech should sound (“A lady whispers to her friend slowly”) and give the text to speak; the model produces a mel spectrogram in that style.

How It Works

The paper’s model (§2): a style encoder, BERT-base whose [CLS] state is the style representation, fine-tuned with P-tuning v2 on an auxiliary task that predicts the prompt’s gender, pitch, speaking speed, volume and emotion classes; a content encoder, FastSpeech 2’s phoneme encoder of 4 Transformer blocks topped by its variance adaptor (duration, pitch, energy), with the style representation prepended to the input of every block; and a speech decoder of the same blocks, with the style prepended to each block’s input, that produces the mel spectrogram (a HiFi-GAN vocoder turns it into audio). Style and content representations are 256 wide (§4.1), so a linear map takes BERT’s 768-wide [CLS] state to 256.

Training is in two stages (CurrentStage): StyleClassification tunes the style encoder on the style-factor labels (TrainStyle), then Synthesis trains the rest on FastSpeech 2’s objective from samples carrying the phonemes, the forced-alignment durations and the style prompt (Description).

Constructors

Constructor Summary
PromptTTS(NeuralNetworkArchitecture<T> architecture, PromptTTSOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a native, trainable PromptTTS.
PromptTTS(NeuralNetworkArchitecture<T> architecture, string modelPath, PromptTTSOptions options) Creates a native, trainable PromptTTS.

Properties

Property Summary
PromptTTSTrainingStage CurrentStage { get; } The network Train updates; the style encoder trains through TrainStyle.
int FftSize { get; }
int HopSize { get; }
int MaxTextLength { get; }
int MelChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
T EvaluateStyleLoss(Tensor<T> stylePrompt, int[] factorLabels) The style-classification loss of a prompt and its labels, without dropout or an update.
T TrainStyle(Tensor<T> stylePrompt, int[] factorLabels) One step of the auxiliary style-classification task (§2.2): the summed cross-entropy of each style factor’s label (StyleFactorClasses order) given the style prompt’s ids.
Tensor<T> TextToMel(string text) Generates a mel spectrogram from text in the voice’s style (pair it with a vocoder).
TtsVoice<T> CreateVoice(string stylePrompt) A voice for Voice: the style prompt (“A lady whispers to her friend slowly”).
int[] PredictStyleFactors(Tensor<T> stylePrompt) The most likely class of each style factor for a style prompt.
int[] TokenizeStyle(string stylePrompt) The style prompt’s bert-base-uncased token ids, [CLS] … [SEP].
void LoadStyleEncoderWeights(string path) Loads a Hugging Face BERT checkpoint (bert-base-uncased’s BertModel or a bert.-prefixed one) into the style encoder, with its pooler.