StyleTTS<T>
Models & Types · AiDotNet.TextToSpeech.StyleEmotion
StyleTTS: a style-based generative model that speaks text in the style of a reference recording (Li et al. 2022).
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → StyleTTS<T>
Implements: IAcousticModel<T>
For Beginners
Give the model a short recording of how you want the speech to sound and the text to say; it copies the speaking style — pitch movement, speed, emotion — onto the new sentence.
How It Works
The paper’s eight modules (§II-A, Appendix D; reference yl4579/StyleTTS models.py): a text encoder (three
convolutions and a BiLSTM over phonemes); a style encoder (four residual blocks and average pooling over a mel
spectrogram, giving a 128-dimensional style); a decoder of seven residual blocks whose AdaIN layers read the style and
which also reads the aligned phoneme features, pitch and energy; a duration predictor and a prosody predictor (pitch
and energy) conditioned on the style through AdaLN and AdaIN; and, for training only, a discriminator, a text aligner
(the transferable monotonic aligner, pre-trained for ASR) and a pitch extractor (a JDC network). The output is a mel
spectrogram; a 24 kHz HiFi-GAN turns it into audio.
Training has two stages (§II-B; CurrentStage): reconstruction of the mel spectrogram from the aligned
phonemes, extracted pitch and energy and the style, adversarially and with the aligner’s objectives; then the duration
and prosody predictors, on recordings stretched in time. Synthesis reads the style from a reference recording
(Voice).
Constructors
| Constructor | Summary |
|---|---|
StyleTTS(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
StyleTTS(NeuralNetworkArchitecture<T> architecture, StyleTTSOptions options) |
Creates a native, trainable StyleTTS. |
StyleTTS(NeuralNetworkArchitecture<T> architecture, string modelPath, StyleTTSOptions options) |
Creates an ONNX-backed StyleTTS for inference. |
Properties
| Property | Summary |
|---|---|
StyleTtsTrainingStage CurrentStage { get; set; } |
The modules a training step updates. |
bool TrainsOnSampleObjective { get; } |
|
int FftSize { get; } |
|
int HopSize { get; } |
|
int MaxTextLength { get; } |
|
int MelChannels { get; } |
|
int ReconstructionUpdates { get; set; } |
Reconstruction updates taken (they decide when the aligner’s objectives start). |
int SkippedSamples { get; set; } |
Training samples skipped because their segment was shorter than MinTrainingFrames, as the reference skips them. |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> ComputeMel(Tensor<T> audio) |
The model’s mel spectrogram [mels, frames] of a waveform at its sample rate: torchaudio’s MelSpectrogram (centred STFT, periodic Hann window, power 2, HTK mel scale), then (log(1e-5 + mel) − mean) / std, as the reference preprocesses. |
Tensor<T> ComputeStyle(Tensor<T> referenceMel) |
The style vector [1, style] of a reference mel spectrogram [frames, mels]. |
Tensor<T> TextToMel(string text) |
The mel spectrogram [frames, mels] of text in the voice’s style (pair it with a 24 kHz vocoder). |
TtsVoice<T> CreateVoice(Tensor<T> recording) |
A voice for Voice: a reference recording at the model’s sample rate whose speaking style the synthesized speech takes. |
int[] EncodePhonemes(string phonemes) |
The symbol ids of IPA phonemes as espeak writes them (words separated by spaces), framed by the pad symbol as the reference frames them; characters outside the table are skipped, as the reference skips them. |
void LoadAlignerWeights(string path) |
Loads the released text aligner (Utils/ASR/epoch_00080.pth’s model entry, saved as safetensors or .pt). |
void LoadPitchExtractorWeights(string path) |
Loads the released pitch extractor (Utils/JDC/bst.t7’s net entry, saved as safetensors or .pt). |
void LoadReferenceWeights(string path, bool includeDiscriminator) |
Loads a StyleTTS checkpoint flattened to named tensors (safetensors, or a .pt state dict) whose names are the reference’s module keys (text_encoder, style_encoder, decoder, predictor, text_aligner, pitch_extractor and optional… |
Fields
| Field | Summary |
|---|---|
int SymbolCount |
The number of entries in the released symbol table (178). |