PromptTTS<T>
Models & Types · AiDotNet.TextToSpeech.DescriptionBased
PromptTTS: speech whose style a plain-language prompt describes (Guo et al. 2023).
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VarianceAdaptorTtsModelBase<T> → PromptTTS<T>
Implements: IAcousticModel<T>
For Beginners
You describe how the speech should sound (“A lady whispers to her friend slowly”) and give the text to speak; the model produces a mel spectrogram in that style.
How It Works
The paper’s model (§2): a style encoder, BERT-base whose [CLS] state is the style representation, fine-tuned with P-tuning v2 on an auxiliary task that predicts the prompt’s gender, pitch, speaking speed, volume and emotion classes; a content encoder, FastSpeech 2’s phoneme encoder of 4 Transformer blocks topped by its variance adaptor (duration, pitch, energy), with the style representation prepended to the input of every block; and a speech decoder of the same blocks, with the style prepended to each block’s input, that produces the mel spectrogram (a HiFi-GAN vocoder turns it into audio). Style and content representations are 256 wide (§4.1), so a linear map takes BERT’s 768-wide [CLS] state to 256.
Training is in two stages (CurrentStage): StyleClassification
tunes the style encoder on the style-factor labels (TrainStyle), then
Synthesis trains the rest on FastSpeech 2’s objective from samples carrying the
phonemes, the forced-alignment durations and the style prompt (Description).
Constructors
| Constructor | Summary |
|---|---|
PromptTTS(NeuralNetworkArchitecture<T> architecture, PromptTTSOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a native, trainable PromptTTS. |
PromptTTS(NeuralNetworkArchitecture<T> architecture, string modelPath, PromptTTSOptions options) |
Creates a native, trainable PromptTTS. |
Properties
| Property | Summary |
|---|---|
PromptTTSTrainingStage CurrentStage { get; } |
The network Train updates; the style encoder trains through TrainStyle. |
int FftSize { get; } |
|
int HopSize { get; } |
|
int MaxTextLength { get; } |
|
int MelChannels { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateStyleLoss(Tensor<T> stylePrompt, int[] factorLabels) |
The style-classification loss of a prompt and its labels, without dropout or an update. |
T TrainStyle(Tensor<T> stylePrompt, int[] factorLabels) |
One step of the auxiliary style-classification task (§2.2): the summed cross-entropy of each style factor’s label (StyleFactorClasses order) given the style prompt’s ids. |
Tensor<T> TextToMel(string text) |
Generates a mel spectrogram from text in the voice’s style (pair it with a vocoder). |
TtsVoice<T> CreateVoice(string stylePrompt) |
A voice for Voice: the style prompt (“A lady whispers to her friend slowly”). |
int[] PredictStyleFactors(Tensor<T> stylePrompt) |
The most likely class of each style factor for a style prompt. |
int[] TokenizeStyle(string stylePrompt) |
The style prompt’s bert-base-uncased token ids, [CLS] … [SEP]. |
void LoadStyleEncoderWeights(string path) |
Loads a Hugging Face BERT checkpoint (bert-base-uncased’s BertModel or a bert.-prefixed one) into the style encoder, with its pooler. |