StyleTTS2<T>
Models & Types · AiDotNet.TextToSpeech.StyleEmotion
StyleTTS 2: text-to-speech through style diffusion and adversarial training with large speech language models (Li et al. 2023).
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → StyleTTS2<T>
Implements: IEndToEndTts<T>
For Beginners
StyleTTS 2 reads text and speaks it, imagining a fitting speaking style for the text with a diffusion model; given a short recording, it leans that style toward the recorded speaker.
How It Works
The paper’s modules (§3; reference yl4579/StyleTTS2 models.py): StyleTTS’s text encoder and two of its style
encoders, one for the acoustic style the decoder reads and one for the prosodic style the predictors read; PL-BERT, a
phoneme-level ALBERT, whose features feed the prosody predictor (through a linear map) and condition the style
diffusion; a prosody predictor whose duration head predicts 50 bins whose sigmoids sum to the duration; a style
diffusion model — a transformer denoiser with EDM preconditioning, sampled with ancestral DPM-2 — that samples both
styles from the text (and, for the multispeaker model, a reference style); and a decoder that ends in a HiFi-GAN or
iSTFTNet waveform generator with a harmonic-plus-noise source from the predicted pitch. Training also uses the
pre-trained text aligner and pitch extractor, multi-period and multi-resolution discriminators, and a WavLM
discriminator.
Synthesis follows the released inference notebooks: the sampled styles are mixed with the reference’s
(Alpha, Beta) and the waveform is returned directly.
Constructors
| Constructor | Summary |
|---|---|
StyleTTS2(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
StyleTTS2(NeuralNetworkArchitecture<T> architecture, StyleTTS2Options options) |
Creates a native, trainable StyleTTS 2. |
StyleTTS2(NeuralNetworkArchitecture<T> architecture, string modelPath, StyleTTS2Options options) |
Creates an ONNX-backed StyleTTS 2 for inference. |
Properties
| Property | Summary |
|---|---|
StyleTts2TrainingStage CurrentStage { get; set; } |
The modules a training step updates. |
bool TrainsOnSampleObjective { get; } |
|
int HiddenDim { get; } |
|
int JointUpdates { get; set; } |
Stage-2 updates taken (they decide when the diffusion, discriminators and joint training start). |
int MaxTextLength { get; } |
|
int NumFlowSteps { get; } |
|
int PretrainingUpdates { get; set; } |
Stage-1 updates taken (they decide when the aligner’s objectives start). |
int SkippedSamples { get; set; } |
Training samples skipped because their segment was shorter than MinTrainingFrames, as the reference skips them. |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> ComputeMel(Tensor<T> audio) |
The model’s normalized log-mel spectrogram [mels, frames] of a waveform at its sample rate (as StyleTTS computes it). |
Tensor<T> ComputeStyle(Tensor<T> referenceMel) |
The reference style [1, 2 · style] — acoustic then prosodic — of a mel spectrogram [frames, mels] (the notebooks’ compute_style). |
TtsVoice<T> CreateVoice(Tensor<T> recording) |
A voice for Voice: a reference recording at the model’s sample rate (the reference notebooks trim its silences first) whose speaker the synthesized speech takes. |
int[] EncodePhonemes(string phonemes) |
The symbol ids of IPA phonemes as espeak writes them (words separated by spaces), preceded by the pad symbol as the reference’s inference prepends it; characters outside the table are skipped. |
void LoadAlignerWeights(string path) |
Loads the released text aligner (Utils/ASR/epoch_00080.pth’s model entry). |
void LoadBertWeights(string path) |
Loads the released PL-BERT (Utils/PLBERT/step_1000000.t7’s net entry with the module.encoder. prefix removed, as load_plbert does, saved as safetensors or .pt). |
void LoadPitchExtractorWeights(string path) |
Loads the released pitch extractor (Utils/JDC/bst.t7’s net entry). |
void LoadReferenceWeights(string path, bool includeDiscriminators) |
Loads the generator’s modules from source and, given theirs, the discriminators. |
void LoadSlmWeights(string path) |
Loads WavLM (a transformers WavLMModel state dict such as microsoft/wavlm-base-plus, safetensors or .pt; the position convolution’s weight norm under either its old weight_g/weight_v names or the parametrized ones). |