StyleTTSZS<T>

Models & Types · AiDotNet.TextToSpeech.StyleEmotion

StyleTTS-ZS: efficient high-quality zero-shot text-to-speech with distilled time-varying style diffusion (Li et al. 2024).

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → StyleTTSZS<T>

Implements: IEndToEndTts<T>

For Beginners

Give StyleTTS-ZS a few seconds of anyone’s speech and some text; it speaks the text in that voice, choosing a natural rhythm and intonation in one fast step.

How It Works

The paper’s four modules (§III): an acoustic synthesizer — a joint prompt-text encoder (conformer blocks over the phonemes and the prompt’s mel-spectrogram, giving prompt-aligned text embeddings and a pooled global style) and StyleTTS 2’s decoder, which reads them with the phoneme durations, pitch and energy; a vector-quantized prosody autoencoder — a prosody encoder compressing durations (as upsampled position embeddings), pitch and energy into a fixed-length time-varying style (50 rows, 9 quantizer codebooks), and duration and pitch-energy decoders conditioned on PL-BERT’s phoneme embeddings; a style diffusion denoiser sampling that style from the text and the prompt (v-objective, angular schedule, DDIM, classifier-free guidance) and a one-step student distilled from it; and multimodal discriminators for the waveform (on WavLM features) and the prosody, besides StyleTTS 2’s multi-period and multi-resolution ones.

No code was released; where the paper is silent the model follows StyleTTS 2’s code and the components the paper cites (docs/text-to-speech/paper-fidelity-decisions.md lists each choice).

Constructors

Constructor Summary
StyleTTSZS(NeuralNetworkArchitecture<T> architecture) Calls the fully specified overload with the omitted parameters’ defaults.
StyleTTSZS(NeuralNetworkArchitecture<T> architecture, StyleTTSZSOptions options) Creates a native, trainable StyleTTS-ZS.
StyleTTSZS(NeuralNetworkArchitecture<T> architecture, string modelPath, StyleTTSZSOptions options) Creates an ONNX-backed StyleTTS-ZS for inference.

Properties

Property Summary
StyleTtsZsTrainingStage CurrentStage { get; set; } The modules a training step updates.
bool TrainsOnSampleObjective { get; }
int HiddenDim { get; }
int MaxTextLength { get; }
int NumFlowSteps { get; }
int SkippedSamples { get; set; } Samples skipped because their segment was shorter than MinTrainingFrames.
int TrainingUpdates { get; set; } Training updates taken (they decide when the aligner’s objectives and the adversarial terms start).

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> ComputeMel(Tensor<T> audio) The normalized log-mel spectrogram [mels, frames] of a waveform (StyleTTS’s front end).
TtsVoice<T> CreateVoice(Tensor<T> recording) A voice: a few seconds of the target speaker at the model’s sample rate (the evaluation uses 3 s).
int[] EncodePhonemes(string phonemes) The symbol ids of IPA phonemes, framed by the pad symbol as StyleTTS 2’s data frames them.
void LoadAlignerWeights(string path) Loads StyleTTS 2’s released text aligner (Utils/ASR/epoch_00080.pth’s model entry).
void LoadBertWeights(string path) Loads the released PL-BERT (yl4579/PL-BERT’s step_1000000.t7, the module.encoder. prefix removed).
void LoadPitchExtractorWeights(string path) Loads StyleTTS 2’s released pitch extractor (Utils/JDC/bst.t7’s net entry).
void LoadSlmWeights(string path) Loads WavLM-base-plus (a transformers WavLMModel state dict).
void LoadSpeakerWeights(string path) Loads a WeSpeaker ResNet34 (avg_model.pt, e.g.