StyleTTS<T>

Models & Types · AiDotNet.TextToSpeech.StyleEmotion

StyleTTS: a style-based generative model that speaks text in the style of a reference recording (Li et al. 2022).

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → StyleTTS<T>

Implements: IAcousticModel<T>

For Beginners

Give the model a short recording of how you want the speech to sound and the text to say; it copies the speaking style — pitch movement, speed, emotion — onto the new sentence.

How It Works

The paper’s eight modules (§II-A, Appendix D; reference yl4579/StyleTTS models.py): a text encoder (three convolutions and a BiLSTM over phonemes); a style encoder (four residual blocks and average pooling over a mel spectrogram, giving a 128-dimensional style); a decoder of seven residual blocks whose AdaIN layers read the style and which also reads the aligned phoneme features, pitch and energy; a duration predictor and a prosody predictor (pitch and energy) conditioned on the style through AdaLN and AdaIN; and, for training only, a discriminator, a text aligner (the transferable monotonic aligner, pre-trained for ASR) and a pitch extractor (a JDC network). The output is a mel spectrogram; a 24 kHz HiFi-GAN turns it into audio.

Training has two stages (§II-B; CurrentStage): reconstruction of the mel spectrogram from the aligned phonemes, extracted pitch and energy and the style, adversarially and with the aligner’s objectives; then the duration and prosody predictors, on recordings stretched in time. Synthesis reads the style from a reference recording (Voice).

Constructors

Constructor Summary
StyleTTS(NeuralNetworkArchitecture<T> architecture) Calls the fully specified overload with the omitted parameters’ defaults.
StyleTTS(NeuralNetworkArchitecture<T> architecture, StyleTTSOptions options) Creates a native, trainable StyleTTS.
StyleTTS(NeuralNetworkArchitecture<T> architecture, string modelPath, StyleTTSOptions options) Creates an ONNX-backed StyleTTS for inference.

Properties

Property Summary
StyleTtsTrainingStage CurrentStage { get; set; } The modules a training step updates.
bool TrainsOnSampleObjective { get; }
int FftSize { get; }
int HopSize { get; }
int MaxTextLength { get; }
int MelChannels { get; }
int ReconstructionUpdates { get; set; } Reconstruction updates taken (they decide when the aligner’s objectives start).
int SkippedSamples { get; set; } Training samples skipped because their segment was shorter than MinTrainingFrames, as the reference skips them.

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> ComputeMel(Tensor<T> audio) The model’s mel spectrogram [mels, frames] of a waveform at its sample rate: torchaudio’s MelSpectrogram (centred STFT, periodic Hann window, power 2, HTK mel scale), then (log(1e-5 + mel) − mean) / std, as the reference preprocesses.
Tensor<T> ComputeStyle(Tensor<T> referenceMel) The style vector [1, style] of a reference mel spectrogram [frames, mels].
Tensor<T> TextToMel(string text) The mel spectrogram [frames, mels] of text in the voice’s style (pair it with a 24 kHz vocoder).
TtsVoice<T> CreateVoice(Tensor<T> recording) A voice for Voice: a reference recording at the model’s sample rate whose speaking style the synthesized speech takes.
int[] EncodePhonemes(string phonemes) The symbol ids of IPA phonemes as espeak writes them (words separated by spaces), framed by the pad symbol as the reference frames them; characters outside the table are skipped, as the reference skips them.
void LoadAlignerWeights(string path) Loads the released text aligner (Utils/ASR/epoch_00080.pth’s model entry, saved as safetensors or .pt).
void LoadPitchExtractorWeights(string path) Loads the released pitch extractor (Utils/JDC/bst.t7’s net entry, saved as safetensors or .pt).
void LoadReferenceWeights(string path, bool includeDiscriminator) Loads a StyleTTS checkpoint flattened to named tensors (safetensors, or a .pt state dict) whose names are the reference’s module keys (text_encoder, style_encoder, decoder, predictor, text_aligner, pitch_extractor and optional…

Fields

Field Summary
int SymbolCount The number of entries in the released symbol table (178).