StyleTTS2<T>

Models & Types · AiDotNet.TextToSpeech.StyleEmotion

StyleTTS 2: text-to-speech through style diffusion and adversarial training with large speech language models (Li et al. 2023).

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → StyleTTS2<T>

Implements: IEndToEndTts<T>

For Beginners

StyleTTS 2 reads text and speaks it, imagining a fitting speaking style for the text with a diffusion model; given a short recording, it leans that style toward the recorded speaker.

How It Works

The paper’s modules (§3; reference yl4579/StyleTTS2 models.py): StyleTTS’s text encoder and two of its style encoders, one for the acoustic style the decoder reads and one for the prosodic style the predictors read; PL-BERT, a phoneme-level ALBERT, whose features feed the prosody predictor (through a linear map) and condition the style diffusion; a prosody predictor whose duration head predicts 50 bins whose sigmoids sum to the duration; a style diffusion model — a transformer denoiser with EDM preconditioning, sampled with ancestral DPM-2 — that samples both styles from the text (and, for the multispeaker model, a reference style); and a decoder that ends in a HiFi-GAN or iSTFTNet waveform generator with a harmonic-plus-noise source from the predicted pitch. Training also uses the pre-trained text aligner and pitch extractor, multi-period and multi-resolution discriminators, and a WavLM discriminator.

Synthesis follows the released inference notebooks: the sampled styles are mixed with the reference’s (Alpha, Beta) and the waveform is returned directly.

Constructors

Constructor Summary
StyleTTS2(NeuralNetworkArchitecture<T> architecture) Calls the fully specified overload with the omitted parameters’ defaults.
StyleTTS2(NeuralNetworkArchitecture<T> architecture, StyleTTS2Options options) Creates a native, trainable StyleTTS 2.
StyleTTS2(NeuralNetworkArchitecture<T> architecture, string modelPath, StyleTTS2Options options) Creates an ONNX-backed StyleTTS 2 for inference.

Properties

Property Summary
StyleTts2TrainingStage CurrentStage { get; set; } The modules a training step updates.
bool TrainsOnSampleObjective { get; }
int HiddenDim { get; }
int JointUpdates { get; set; } Stage-2 updates taken (they decide when the diffusion, discriminators and joint training start).
int MaxTextLength { get; }
int NumFlowSteps { get; }
int PretrainingUpdates { get; set; } Stage-1 updates taken (they decide when the aligner’s objectives start).
int SkippedSamples { get; set; } Training samples skipped because their segment was shorter than MinTrainingFrames, as the reference skips them.

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> ComputeMel(Tensor<T> audio) The model’s normalized log-mel spectrogram [mels, frames] of a waveform at its sample rate (as StyleTTS computes it).
Tensor<T> ComputeStyle(Tensor<T> referenceMel) The reference style [1, 2 · style] — acoustic then prosodic — of a mel spectrogram [frames, mels] (the notebooks’ compute_style).
TtsVoice<T> CreateVoice(Tensor<T> recording) A voice for Voice: a reference recording at the model’s sample rate (the reference notebooks trim its silences first) whose speaker the synthesized speech takes.
int[] EncodePhonemes(string phonemes) The symbol ids of IPA phonemes as espeak writes them (words separated by spaces), preceded by the pad symbol as the reference’s inference prepends it; characters outside the table are skipped.
void LoadAlignerWeights(string path) Loads the released text aligner (Utils/ASR/epoch_00080.pth’s model entry).
void LoadBertWeights(string path) Loads the released PL-BERT (Utils/PLBERT/step_1000000.t7’s net entry with the module.encoder. prefix removed, as load_plbert does, saved as safetensors or .pt).
void LoadPitchExtractorWeights(string path) Loads the released pitch extractor (Utils/JDC/bst.t7’s net entry).
void LoadReferenceWeights(string path, bool includeDiscriminators) Loads the generator’s modules from source and, given theirs, the discriminators.
void LoadSlmWeights(string path) Loads WavLM (a transformers WavLMModel state dict such as microsoft/wavlm-base-plus, safetensors or .pt; the position convolution’s weight norm under either its old weight_g/weight_v names or the parametrized ones).