StyleTTSZS<T>
Models & Types · AiDotNet.TextToSpeech.StyleEmotion
StyleTTS-ZS: efficient high-quality zero-shot text-to-speech with distilled time-varying style diffusion (Li et al. 2024).
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → StyleTTSZS<T>
Implements: IEndToEndTts<T>
For Beginners
Give StyleTTS-ZS a few seconds of anyone’s speech and some text; it speaks the text in that voice, choosing a natural rhythm and intonation in one fast step.
How It Works
The paper’s four modules (§III): an acoustic synthesizer — a joint prompt-text encoder (conformer blocks over the phonemes and the prompt’s mel-spectrogram, giving prompt-aligned text embeddings and a pooled global style) and StyleTTS 2’s decoder, which reads them with the phoneme durations, pitch and energy; a vector-quantized prosody autoencoder — a prosody encoder compressing durations (as upsampled position embeddings), pitch and energy into a fixed-length time-varying style (50 rows, 9 quantizer codebooks), and duration and pitch-energy decoders conditioned on PL-BERT’s phoneme embeddings; a style diffusion denoiser sampling that style from the text and the prompt (v-objective, angular schedule, DDIM, classifier-free guidance) and a one-step student distilled from it; and multimodal discriminators for the waveform (on WavLM features) and the prosody, besides StyleTTS 2’s multi-period and multi-resolution ones.
No code was released; where the paper is silent the model follows StyleTTS 2’s code and the components the paper cites (docs/text-to-speech/paper-fidelity-decisions.md lists each choice).
Constructors
| Constructor | Summary |
|---|---|
StyleTTSZS(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
StyleTTSZS(NeuralNetworkArchitecture<T> architecture, StyleTTSZSOptions options) |
Creates a native, trainable StyleTTS-ZS. |
StyleTTSZS(NeuralNetworkArchitecture<T> architecture, string modelPath, StyleTTSZSOptions options) |
Creates an ONNX-backed StyleTTS-ZS for inference. |
Properties
| Property | Summary |
|---|---|
StyleTtsZsTrainingStage CurrentStage { get; set; } |
The modules a training step updates. |
bool TrainsOnSampleObjective { get; } |
|
int HiddenDim { get; } |
|
int MaxTextLength { get; } |
|
int NumFlowSteps { get; } |
|
int SkippedSamples { get; set; } |
Samples skipped because their segment was shorter than MinTrainingFrames. |
int TrainingUpdates { get; set; } |
Training updates taken (they decide when the aligner’s objectives and the adversarial terms start). |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> ComputeMel(Tensor<T> audio) |
The normalized log-mel spectrogram [mels, frames] of a waveform (StyleTTS’s front end). |
TtsVoice<T> CreateVoice(Tensor<T> recording) |
A voice: a few seconds of the target speaker at the model’s sample rate (the evaluation uses 3 s). |
int[] EncodePhonemes(string phonemes) |
The symbol ids of IPA phonemes, framed by the pad symbol as StyleTTS 2’s data frames them. |
void LoadAlignerWeights(string path) |
Loads StyleTTS 2’s released text aligner (Utils/ASR/epoch_00080.pth’s model entry). |
void LoadBertWeights(string path) |
Loads the released PL-BERT (yl4579/PL-BERT’s step_1000000.t7, the module.encoder. prefix removed). |
void LoadPitchExtractorWeights(string path) |
Loads StyleTTS 2’s released pitch extractor (Utils/JDC/bst.t7’s net entry). |
void LoadSlmWeights(string path) |
Loads WavLM-base-plus (a transformers WavLMModel state dict). |
void LoadSpeakerWeights(string path) |
Loads a WeSpeaker ResNet34 (avg_model.pt, e.g. |