Text To Speech

418 public types in this namespace, organized by kind.

Models & Types (151)

Type Summary
APNet2<T> APNet2: a high-quality, high-efficiency vocoder that predicts the amplitude and phase spectra directly with ConvNeXt v2 networks and reconstructs the waveform by an inverse STFT, trained with linear anti-wrapping phase losses and a hinge GA…
APNet<T> APNet: an all-frame-level neural vocoder that predicts the log amplitude spectrum (ASP) and the phase spectrum (PSP) directly from the mel spectrogram with residual convolution networks and reconstructs the waveform by an inverse STFT.
AcousticConditioning<T> The phoneme hidden sequence after a variance-adaptor model adds its acoustic conditions, with what its decoder is conditioned on and any auxiliary loss the conditioning contributes in training.
AcousticTargets<T> An utterance’s acoustic training targets, aligned to its mel frames.
AdaSpeech2<T> AdaSpeech 2: adaptive TTS that adapts to a new voice from untranscribed speech, through a mel-spectrogram encoder aligned to the phoneme encoder’s output space.
AdaSpeech<T> AdaSpeech: adaptive TTS for custom voice with acoustic condition modeling and conditional layer normalization.
AlignTTS<T> AlignTTS: alignment-free non-autoregressive TTS using a feed-forward Transformer and mix density network alignment.
AmazonPolly<T> A client of Amazon Polly’s SynthesizeSpeech API.
AudioPaLM<T> AudioPaLM (Rubenstein et al.
AzureNeuralTTS<T> A client of the Azure AI Speech text-to-speech REST API.
BarkGenerationResult<T> Structured Bark output, including every intermediate representation.
BarkHistoryPrompt Optional semantic and acoustic history used to preserve a Bark voice/prompt.
BarkModel<T> Low-level, checkpoint-faithful Bark foundation model.
Bark<T> Beginner-friendly text façade over the shared BarkModel foundation model.
BigVGAN<T> BigVGAN: a universal GAN vocoder — HiFi-GAN with anti-aliased periodic (Snake) activations, trained at scale against multi-period and multi-resolution discriminators.
CSM<T> CSM, Sesame’s Conversational Speech Model (Sesame 2025, “Crossing the uncanny valley of conversational voice”): a Llama backbone reads the conversation — text and Mimi’s audio codes interleaved, one frame per position — and predicts each ne…
ChatTTS<T> ChatTTS (2noise, 2024): conversational text-to-speech as a GPT over text and DVAE audio codes, with prosody control tokens, a speaker embedding, and Vocos rendering.
Chatterbox<T> Resemble AI’s Chatterbox: an emotion-controllable zero-shot T3 model whose speech tokens are rendered by S3Gen.
CoMoSpeech<T> CoMoSpeech: one-step speech synthesis with a consistency model distilled from an EDM diffusion teacher.
CosyVoice2<T> CosyVoice 2: streaming TTS with a pretrained LLM, finite-scalar-quantized speech tokens and chunk-aware causal flow matching.
CosyVoice3<T> Fun-CosyVoice 3: Qwen2 speech-token generation, S3-v3 prompt tokens, causal DiT flow matching and CausalHiFT.
CosyVoiceClone<T> Voice-cloning facade over the released CosyVoice supervised-token, conditional-flow-matching, and HiFT pipeline.
CosyVoiceToken2TextReward<T> A trainable token-to-text sequence model that supplies a differentiable ASR reward for DiffRO.
CosyVoice<T> CosyVoice: multilingual TTS with supervised semantic tokens and conditional flow matching.
CsmSegment<T> A turn of conversation CSM conditions on: who spoke (Speaker), what they said (Text), and the recording of it (Audio, at the model’s 24 kHz).
DeepVoice3<T> Deep Voice 3: fully convolutional attention-based text-to-speech.
DiTToCheckpointPaths Files used to assemble DiTTo-TTS.
DiTToTTS<T> DiTTo-TTS: scalable zero-shot text-to-speech with a diffusion Transformer in CLaM-TTS Mel-VAE space.
Dia<T> Dia, Nari Labs’ dialogue model (2025): a Transformer encoder reads the script’s UTF-8 bytes — speaker tags [S1] and [S2] as bytes 1 and 2, non-verbal cues such as “(laughs)” as plain text — and an encoder-decoder Transformer predicts DA…
DiffWave<T> DiffWave: a versatile diffusion model for audio synthesis — a non-autoregressive noise predictor built from bidirectional dilated convolutions, conditioned on the mel spectrogram, sampled by the reverse diffusion process.
E2TTS<T> E2 TTS (“Embarrassingly Easy”): fully non-autoregressive text-to-speech by flow-matching speech infilling over characters padded with filler tokens to the speech length, with a flat U-Net Transformer.
E3TTSCheckpointPaths Checkpoint inputs for E3’s public BERT dependency and AiDotNet-converted private U-Net weights.
E3TTS<T> E3 TTS: BERT-conditioned, raw-waveform diffusion without acoustic intermediates or explicit alignment.
ElevenLabsTTS<T> A client of the ElevenLabs text-to-speech API.
EmotiVoiceFrontEnd EmotiVoice’s bilingual text front end (netease-youdao/EmotiVoice frontend.py, frontend_cn.py, frontend_en.py): Chinese and English text to the released model’s phoneme tokens.
EmotiVoice<T> EmotiVoice: NetEase Youdao’s multi-voice, prompt-controlled emotional text-to-speech (2023).
EnglishG2P An English grapheme-to-phoneme front end that writes text as espeak-ng’s American English phonemes, in the symbol sequence phonemizer produces for the TTS models trained on them (Pheme, VALL-E, and others built on USLM’s symbol table).
F5TTS<T> F5-TTS: a fully non-autoregressive text-to-speech model that fills in speech with flow matching, using a Diffusion Transformer and ConvNeXt V2 text refinement over characters padded to the speech length.
FastSpeech2<T> FastSpeech 2: non-autoregressive TTS with a variance adaptor for duration, pitch and energy.
FastSpeech<T> FastSpeech: non-autoregressive TTS with knowledge-distilled duration predictor for parallel generation.
FireRedTTS<T> FireRedTTS (Guo et al.
FishSpeechV15<T> Fish Speech 1.5 (Liao et al.
FishSpeech<T> Fish Speech 1.4: the Dual-AR transformer and Firefly-GAN of FishSpeechV15 (Liao et al.
ForwardTacotron<T> ForwardTacotron: a non-autoregressive Tacotron that predicts phoneme durations, pitch and energy and generates the whole mel spectrogram in one pass.
FreGrad<T> FreGrad: a lightweight and fast frequency-aware diffusion vocoder that denoises the waveform’s two Haar wavelet sub-bands with frequency-aware dilated convolutions.
GLM4Voice<T> GLM-4-Voice (Zeng et al.
GPTSoVITS<T> GPT-SoVITS (RVC-Boss, 2024): few-shot voice cloning in which a GPT writes 25 Hz semantic tokens for the text, continuing a reference recording’s, and a synthesizer speaks them in the reference voice: SoVITS’s VITS-style GAN for v1 to v2ProP…
GlmVoiceDialogueTurn One conversational turn for GLM-4-Voice’s supervised fine-tuning (§4.2): the user’s speech or text, and the reply’s text and speech tokens.
GlmVoiceReply<T> A GLM-4-Voice reply: its text, its speech tokens and its 22.05 kHz audio.
GlowTTS<T> Glow-TTS: a flow-based generative text-to-speech model that finds its own alignment with monotonic alignment search.
GoogleCloudTTS<T> A client of the Google Cloud Text-to-Speech API.
GptSoVitsCheckpoints The files a GPTSoVITS loads (the release’s GPT_SoVITS/pretrained_models, Hugging Face lj1995/GPT-SoVITS).
GradTTS<T> Grad-TTS: text-to-speech with a score-based diffusion decoder that turns an aligned Gaussian prior into a mel spectrogram.
HiFiGAN<T> HiFi-GAN: a GAN vocoder whose generator upsamples a mel spectrogram to a waveform through transposed convolutions and multi-receptive-field residual blocks, trained against multi-period and multi-scale discriminators.
ISTFTNet<T> iSTFTNet: a fast, lightweight mel-spectrogram vocoder — HiFi-GAN with its output-side upsampling replaced by an inverse STFT of a small magnitude and phase spectrogram it predicts.
IndexTTS2<T> IndexTTS2 (Zhou et al.
IndexTTS<T> IndexTTS (Deng et al.
IndexTts2Checkpoints The files IndexTTS2’s release reads (infer_v2.py and ensure_models_available).
KaniTTS2<T> KaniTTS 2 (nineninesix.ai, 2026): Kani’s LFM2 language model writing NVIDIA NanoCodec codes, with frame-level rotary positions, a learnable rotary scale per attention layer and a speaker embedding from a recording (WavLM-Large speaker embed…
KaniTTS<T> KaniTTS (nineninesix.ai, 2025): text-to-speech as language modeling — an LFM2 language model reads the text and writes NVIDIA NanoCodec codes, four per frame at 12.5 frames a second, which NanoCodec’s causal decoder turns into 22.05 kHz aud…
Kokoro<T> Kokoro: lightweight end-to-end TTS with a StyleTTS2-inspired architecture using style tokens and ISTFTNet decoder.
LlamaOmniResponse<T> A LLaMA-Omni response: its text, its speech units and its 16 kHz audio.
LlamaOmni<T> LLaMA-Omni (Fang et al., ICLR 2025): low-latency speech interaction with an LLM.
Llasa<T> Llasa (Ye et al.
MARS5TTS<T> MARS5 (Camb.ai, 2024): two-stage voice cloning over EnCodec’s 6 kbps codes.
MandarinG2P Converts Mandarin Chinese text to IPA phonemes, as VALL-E X’s reference front end does (Plachtaa/VALL-E-X, utils/g2p/mandarin.py, chinese_to_ipa).
Mars5Checkpoints The files a MARS5TTS loads.
MaskGCT<T> MaskGCT (Wang et al.
MatchaTTS<T> Matcha-TTS: a non-autoregressive acoustic model whose decoder is trained with optimal-transport conditional flow matching and samples a mel spectrogram in a few ODE steps.
MegaTTS2<T> Mega-TTS 2 (Jiang et al.
MegaTTS3<T> MegaTTS 3 (Jiang et al.
MegaTTS<T> Mega-TTS (Jiang et al.
MelGAN<T> MelGAN: a non-autoregressive, fully convolutional GAN vocoder — transposed-convolution upsampling with dilated residual stacks — trained against multi-scale window-based discriminators with the hinge loss and feature matching.
MeloTTS<T> MeloTTS (MyShell, myshell-ai/MeloTTS): a multilingual Bert-VITS2 — VITS with VITS2’s speaker-conditioned text encoder, transformer coupling flows, noise-scaled alignment search and duration discriminator, conditioned on per- symbol tones, l…
MetaVoice1B<T> MetaVoice’s learned-position causal GPT, noncausal codebook GPT, speaker LSTM and EnCodec-conditioned four-band diffusion waveform generator.
MinMo<T> MinMo (Chen et al.
MoshiConversation<T> A training example for Moshi’s language model: Moshi’s (the main speaker’s) 24 kHz speech, the user’s if there is one, and the main speaker’s words with their times.
Moshi<T> Moshi (Défossez et al.
MultiBandMelGAN<T> Multi-band MelGAN: a MelGAN generator that predicts PQMF sub-bands of the waveform at a quarter of its rate, trained with full- and sub-band multi-resolution STFT losses and multi-scale discriminators.
Murf<T> A client of the Murf text-to-speech API.
NVIDIARivaTTS<T> A client of an NVIDIA Riva / Speech NIM text-to-speech server’s HTTP API.
NaturalSpeech2<T> NaturalSpeech 2 (Shen et al.
NaturalSpeech3<T> NaturalSpeech 3 (Ju et al.
NaturalSpeechSupPhonemes Sup-phonemes (Mixed-Phoneme BERT, Zhang et al.
NaturalSpeech<T> NaturalSpeech (Tan et al.
NonAttentiveTacotron<T> Non-Attentive Tacotron: Tacotron 2 with its attention replaced by an explicit duration predictor and Gaussian upsampling (supervised-duration variant).
OpenVoiceV2<T> OpenVoiceV2: OpenVoice V2: Improved Instant Voice Cloning.
OpenVoice<T> OpenVoice: versatile instant voice cloning with decoupled tone color conversion.
OrpheusTTS<T> Orpheus (Canopy Labs, 2025): text-to-speech as plain language modeling — Llama 3.2 3B reads “{voice}: {text}” and continues it with SNAC codes, seven tokens per frame, which SNAC’s decoder turns into 24 kHz audio.
OuteTTSFeatures OuteTTS’s audio features of a stretch of speech, each scaled to 0–100.
OuteTTSSpeaker An OuteTTS 1.0 speaker profile (the release’s speaker JSON): the transcript of a short recording, its words with their codes and features, and the whole recording’s features.
OuteTTSWord A word of an OuteTTS speaker profile: its text, its duration, its two DAC codebooks’ codes and its features (each 0–100).
OuteTTS<T> OuteTTS 1.0 (OuteAI): text-to-speech as plain language modeling — a Llama 3.2 or Qwen3 language model reads the text and continues it with each word’s duration, energy, spectral-centroid and pitch tokens and its DAC codes, which a DAC decod…
ParallelWaveGAN<T> Parallel WaveGAN: a non-autoregressive WaveNet that turns Gaussian noise into speech conditioned on a mel spectrogram, trained jointly with a multi-resolution STFT loss and an adversarial loss.
ParlerTTS<T> Parler-TTS: speech in the voice a plain-language description gives (Lyth & King 2024), as the open reproduction huggingface/parler-tts builds it.
Pheme<T> Pheme: efficient and conversational speech generation in two stages over SpeechTokenizer codes, a T5 text-to-semantic model and a non-autoregressive SoundStorm-style semantic-to-acoustic model.
Piper<T> Piper: a fast, local neural text-to-speech system — VITS trained with Rhasspy’s recipe and model sizes.
PortaSpeech<T> PortaSpeech: a portable text-to-speech model with a mixture-alignment linguistic encoder, a VAE variational generator with a normalizing-flow prior, and a flow-based post-net with grouped parameter sharing.
PriorGrad<T> PriorGrad vocoder: DiffWave whose diffusion prior is a data-dependent diagonal Gaussian N(0, Σ_c) with the standard deviation of each frame taken from the mel spectrogram’s normalized frame energy.
ProDiff<T> ProDiff: a progressive fast diffusion model that synthesizes a mel spectrogram in two denoising steps by predicting the clean spectrogram directly and distilling a 4-step teacher.
PromptTTS<T> PromptTTS: speech whose style a plain-language prompt describes (Guo et al.
SPEARTTS<T> SPEAR-TTS: multi-speaker text-to-speech trained with minimal supervision (Kharitonov et al.
SeedTTSClone<T> Voice-cloning facade over Seed-TTS’s audio/text in-context continuation pipeline.
SeedTTS<T> Seed-TTS’s disclosed autoregressive pipeline: speech tokenizer, causal token LM, token diffusion, and acoustic vocoder.
SoundStorm<T> SoundStorm: efficient parallel audio generation (Borsos et al.
SparkTTS<T> Spark-TTS: Qwen2.5 generates a single semantic speech-token stream which BiCodec renders with decoupled speaker tokens.
SpeechGPT<T> SpeechGPT (Zhang et al., Findings of EMNLP 2023): a large language model with intrinsic cross-modal conversational abilities.
SpeechGptResponse<T> A SpeechGPT response: its text and its units, and the speech voiced from them.
SpeechT5Checkpoints The files SpeechT5’s release reads.
SpeechT5<T> SpeechT5 (Ao et al.
SpeedySpeech<T> SpeedySpeech: a fully convolutional non-autoregressive text-to-speech student network trained on durations from a teacher’s attention.
SpiritLM<T> SpiritLM (Nguyen et al., TACL 2025): an interleaved spoken and written language model.
SpiritLmContent<T> A piece of SpiritLM’s input or output: text, or 16 kHz speech.
StepAudioSpeaker<T> A speaker Step-Audio’s TTS speaks as (the release’s speakers_info entry): the name, the prompt recording’s transcript and dual-codebook ids, its 22.05 kHz mel [frames, 80] and CAM++ x-vector.
StepAudio<T> Step-Audio (Huang et al.
StyleTTS2<T> StyleTTS 2: text-to-speech through style diffusion and adversarial training with large speech language models (Li et al.
StyleTTSZS<T> StyleTTS-ZS: efficient high-quality zero-shot text-to-speech with distilled time-varying style diffusion (Li et al.
StyleTTS<T> StyleTTS: a style-based generative model that speaks text in the style of a reference recording (Li et al.
Tacotron2<T> Tacotron 2’s spectrogram prediction network (Shen et al.
TacotronSpectrogram The mel spectrogram and frame energy that Tacotron 2 and the acoustic models following it are trained on.
Tacotron<T> Tacotron: an end-to-end attention-based sequence-to-sequence text-to-speech model with CBHG modules.
TortoiseTTS<T> TorToiSe: Better Speech Synthesis Through Scaling.
TransformerTTS<T> Transformer TTS: Tacotron 2’s text-to-mel pipeline with its recurrent encoder and decoder replaced by a Transformer.
TtsTrainingSample<T> One utterance of text-to-speech training data, with whatever supervision the model’s paper trains on.
TtsVoice<T> The voice a multi-speaker or adaptive text-to-speech model synthesizes in: a speaker and, for models that read one, a reference recording of that speaker.
UniAudio<T> UniAudio: a universal neural codec and multi-scale autoregressive audio Transformer.
UnivNet<T> UnivNet: a GAN vocoder whose generator uses location-variable convolutions with kernels predicted from the mel spectrogram, trained against multi-resolution spectrogram and multi-period waveform discriminators.
VALLE2<T> VALL-E 2: VALL-E with grouped code modeling and repetition-aware sampling, which predicts several first-codebook codes per autoregressive step and avoids the decoding loops of plain sampling.
VALLEXClone<T> Voice-cloning facade over VALL-E X’s cross-lingual neural-codec language model.
VALLEX<T> VALL-E X: VALL-E trained on English and Mandarin with a language ID, which speaks a language in the voice of a prompt recorded in another.
VALLE<T> VALL-E: a neural codec language model for zero-shot text-to-speech, which continues a 3-second recording of an unseen speaker by predicting EnCodec codes from phonemes.
VITS2<T> VITS2: single-stage text-to-speech that improves VITS with an adversarially trained stochastic duration predictor, noise-scaled monotonic alignment search, a Transformer block in the normalizing flows and a speaker-conditioned text encoder.
VITS<T> VITS: end-to-end text-to-speech as a conditional VAE whose prior is a normalizing flow over a text encoding, with a stochastic duration predictor and a HiFi-GAN decoder trained adversarially.
VevoCheckpoints The files Vevo’s release reads (models/vc/vevo/infer_*.py).
Vevo<T> Vevo (Zhang et al., ICLR 2025): controllable zero-shot voice imitation by self-supervised disentanglement — HuBERT’s features quantized into content tokens (32 codes, timbre and style squeezed out) and content-style tokens (8,192 codes, tim…
Vocos<T> Vocos: a GAN vocoder that generates Fourier spectral coefficients — magnitude and phase — at frame rate with a ConvNeXt backbone and turns them into audio with the inverse STFT.
VoiceCraft<T> VoiceCraft: causal-masked, delayed-stacked neural-codec speech infilling and zero-shot TTS.
VoiceFlow<T> VoiceFlow: a text-to-speech acoustic model that generates mel spectrograms with conditional flow matching and straightens its sampling trajectory with rectified flow.
Voicebox<T> Voicebox: a text-guided speech infilling model with separate regression-duration and conditional-flow audio models.
WaveGlow<T> WaveGlow: a flow-based generative network for speech synthesis — Glow’s invertible 1×1 convolutions and affine couplings over groups of audio samples, with WaveNet-like coupling networks conditioned on the mel spectrogram, trained only by m…
WaveGrad<T> WaveGrad: estimating gradients for waveform generation — a diffusion vocoder whose noise predictor is conditioned on the continuous noise level √ᾱ, built from GAN-TTS-style upsampling blocks over the mel spectrogram modulated by FiLM from d…
WaveNet<T> WaveNet: a generative model for raw audio — an autoregressive stack of dilated causal convolutions that predicts a softmax over μ-law-quantized samples, here locally conditioned on the mel spectrogram as a vocoder.
WaveRNN<T> WaveRNN: efficient neural audio synthesis with a single recurrent layer whose state is split to predict the coarse and the fine byte of each 16-bit sample through a dual softmax.
WellSaidLabs<T> A client of the WellSaid Labs text-to-speech API.
WhisperSpeechConditions The cond variant’s conditions; null marks a condition as missing (its own learned code).
WhisperSpeech<T> WhisperSpeech (Collabora 2024): text-to-speech by inverting Whisper, after SPEAR-TTS (Kharitonov et al.
XTTSv2Clone<T> Voice-cloning facade over the canonical XTTS-v2 zero-shot architecture.
XTTSv2<T> XTTS-v2: multilingual zero-shot TTS with a dVAE tokenizer, GPT-2 and speaker-conditioned HiFi-GAN.
YourTTS<T> YourTTS: zero-shot multi-speaker text-to-speech and voice conversion — VITS conditioned on speaker embeddings from a pretrained speaker encoder, with language embeddings for multilingual training.
ZonosConditioner One of Zonos’s conditioners as its release config describes it.
Zonos<T> Zonos, Zyphra’s text-to-speech model (2025): a prefix of conditioning tokens — the text’s eSpeak phonemes, a speaker embedding, and targets for emotion, pitch variation, speaking rate, bandwidth and language — followed by DAC’s nine codeboo…

Base Classes (16)

Type Summary
AcousticModelBase<T> Base class for acoustic TTS models that generate mel-spectrograms from text.
AmplitudePhaseVocoderBase<T> The frame-level amplitude-and-phase vocoders’ shared generator and losses (APNet, Ai and Ling 2023; APNet2, Du et al.
CloudTtsClientBase<T> A client of a hosted text-to-speech service: it sends the text to the service’s REST API and decodes the audio it returns into a waveform tensor.
CosyVoiceBase<T> What the CosyVoice models share (FunAudioLLM/CosyVoice, Apache-2.0): a language model speaking S3 tokens, a flow sampling an 80-band mel from them by conditional flow matching with classifier-free guidance, HiFT turning the mel into audio,…
DiffusionVocoderBase<T> The denoising-diffusion training and sampling diffusion vocoders share (Ho et al.
FishSpeechBase<T> The Fish Speech models (Liao et al.
FlowMatchingTtsModelBase<T> The text-guided speech-infilling conditional flow matching shared by E2 TTS and F5-TTS (Chen et al.
GanVocoderBase<T> The training loop GAN vocoders share: a generator that turns a mel spectrogram into a waveform, discriminators that tell real from generated audio, and alternating discriminator and generator steps on random segments.
KaniTTSBase<T> What both KaniTTS releases share: an LFM2 language model writing NVIDIA NanoCodec codes after the release’s prompt, the codec, the speech extraction, the training sequence and its loss, and loading.
SegmentVocoderBase<T> The training and synthesis plumbing of vocoders trained by one objective on aligned (mel spectrogram, waveform) segments — the diffusion vocoders (DiffWave, WaveGrad, PriorGrad, FreGrad), the flow WaveGlow and the autoregressive WaveNet and…
StochasticDurationVitsBase<T> A VITS-family model whose durations come from VITS’s stochastic duration predictor (Kim et al.
TtsModelBase<T> Base class for text-to-speech neural networks that can operate in both ONNX inference and native training modes.
VallEModelBase<T> What the models of the VALL-E family share: EnCodec, the AR and NAR codec language models (VallECore), a phoneme vocabulary, training each model on its own, and prompted synthesis.
VarianceAdaptorTtsModelBase<T> Base class for non-autoregressive acoustic models of the FastSpeech family: a phoneme encoder, a variance adaptor (duration and length regulation, optionally pitch and energy), and a mel decoder.
VitsTtsModelBase<T> The VITS family’s shared model: a conditional VAE whose prior is a normalizing flow over a text encoding, aligned by monotonic alignment search, with a duration model and a HiFi-GAN decoder trained adversarially on random windows.
VocoderBase<T> Base class for neural vocoder models that convert mel-spectrograms to audio waveforms.

Interfaces (8)

Type Summary
IAcousticModel<T> Interface for acoustic models that generate mel-spectrograms from text.
ICodecTts<T> Interface for codec-based TTS models that use neural audio codecs with language model decoding.
ICosyVoiceTokenReward<T> A differentiable token-domain reward for CosyVoice 3 DiffRO (paper equations 3-7).
IEndToEndTts<T> Interface for end-to-end TTS models that generate waveforms directly from text without a separate vocoder stage.
IStreamingTts<T> Interface for TTS models that support streaming/chunked synthesis with low latency.
ITtsModel<T> Base interface for all text-to-speech models.
IVocoder<T> Interface for neural vocoders that convert mel-spectrograms to audio waveforms.
IVoiceCloner<T> Interface for TTS models that support zero-shot or few-shot voice cloning from reference audio.

Enums (96)

Type Summary
AdaSpeech2TrainingStep The steps of AdaSpeech 2’s adaptation pipeline (Yan et al.
AdaSpeechTrainingPhase The training phases of AdaSpeech (Chen et al.
AlignTTSTrainingPhase The training phases of AlignTTS (Zeng et al.
AudioPalmDetokenizer How AudioPaLM turns audio tokens into speech (§3.3).
AudioPalmSpeechTokenizer How AudioPaLM turns speech into audio tokens (Rubenstein et al.
BarkGenerationStage Identifies a stage in Bark’s semantic-to-waveform generation pipeline.
ChatterboxVariant The released Chatterbox checkpoint a model follows.
CoMoSpeechTrainingPhase The two training stages of CoMoSpeech.
CosyVoiceFlowLoss The loss CosyVoice’s flow matching minimizes.
CosyVoiceTrainingStage Which CosyVoice network a training step trains (the recipes train llm, flow and hifigan in turn).
DiTToModelSize The four diffusion-transformer sizes evaluated by DiTTo-TTS.
DiTToTrainingStage Diffusion and length prediction, plus the separately pretrained and optionally semantically finetuned codec.
DiffWaveConditioner What DiffWave’s noise predictor is conditioned on (Kong et al.
FireRedTtsEmotion An emotion of FireRedTTS’s instruction tuning (§4.2).
FireRedTtsMelDecoder Which of the paper’s Mel decoders turns semantic tokens into the mel (§3.3).
FireRedTtsStage Which of FireRedTTS’s networks a training step updates.
FireRedTtsVocoder Which BigVGAN-v2 vocoder turns the 16 kHz, 10 ms mel into audio.
FishSpeechRecipe Which training recipe a Fish Speech model’s optimizer follows.
GlmVoiceDecoding How GLM-4-Voice’s decoder turns a sequence of speech tokens into audio.
GlmVoiceGeneration How GLM-4-Voice lays out its spoken reply (Zeng et al.
GlmVoiceTrainingStage Which of GLM-4-Voice’s two language-model stages a training step follows (§4).
GlmVoiceTtsFormat How Synthesize reads a text aloud.
GptSoVitsStage Which of GPT-SoVITS’s networks GPTSoVITS trains.
GptSoVitsTextCut How GPT-SoVITS cuts the text before synthesis (how_to_cut).
GptSoVitsTextLanguage How GPT-SoVITS reads the text’s language (inference_webui.dict_language).
GptSoVitsVersion The GPT-SoVITS release a GPTSoVITS follows (each has its own released weights).
IndexTts2DurationConditioning How IndexTTS2’s text-to-semantic model receives the target length.
IndexTts2EmotionMode How IndexTTS2 turns an emotion distribution into the emotion vector.
IndexTts2Stage Which of IndexTTS2’s networks IndexTTS2 trains.
LlamaOmniAlignment Which hidden state stands for a response token when the speech decoder trains.
LlamaOmniTrainingStage Which of LLaMA-Omni’s two training stages (§2.5) a step follows.
Mars5Stage Which of MARS5’s networks MARS5TTS trains.
MaskGctActivation The gated activation in MaskGctTransformer’s feed-forward.
MaskGctDurationMode How MaskGCT chooses the target’s length.
MaskGctStage Which of MaskGCT’s networks MaskGCT trains.
MegaTts2CodeAnchor Where CVQ-VAE’s dynamic initialization draws the anchors that move rarely used codes.
MegaTts2CodeDistance How the CVQ-VAE quantizer of MegaTTS2 finds a vector’s code.
MegaTts2Stage Which of Mega-TTS 2’s training stages MegaTTS2 runs.
MegaTtsLanguage The front end MegaTTS reads text with.
MegaTtsStage Which of Mega-TTS’s two models MegaTTS trains.
MeloLanguage The languages MeloTTS’s checkpoints are trained for (one per checkpoint).
MinMoDuplexAction What the full duplex predictor decides at a position of the conversation.
MinMoTrainingStage MinMo’s training stages (§3.4).
MinMoVoiceEncoder Which voice encoder MinMo reads speech with.
MoshiDepthLayout Which audio streams Moshi’s Depth Transformer predicts.
MoshiTrainingStage Moshi’s training stages (§4.4, Table 1).
MoshiVariant The delay between Moshi’s text and audio streams (§3.4.4, App.
NaturalSpeech2CeResidual The residual each quantizer’s cross-entropy term (L_ce-rvq) is measured on.
NaturalSpeech2Diffusion What NaturalSpeech 2’s latent diffusion learns.
NaturalSpeech2Positions The positional encoding of NaturalSpeech 2’s Transformers.
NaturalSpeech2PromptPosition Where the training prompt is cut from the utterance.
NaturalSpeech2ScoreLoss The form of Eq.
NaturalSpeech2Solver The ODE solver of reverse diffusion.
NaturalSpeech3Stage What NaturalSpeech3’s training step trains.
NaturalSpeechPhonemeEncoder NaturalSpeech’s phoneme encoder.
NaturalSpeechStage NaturalSpeech’s training stages (Tan et al.
NaturalSpeechUpsampling How NaturalSpeech’s learnable upsampling layer is built.
OrpheusDecoding How OrpheusTTS turns the generated codes into audio.
OuteTTSBackbone The language model OuteTTS 1.0 is built on.
PalmDecoderKind Which decoder PalmDecoder builds.
PhemeAcousticDecoding How Pheme’s acoustic model decodes its codebooks at synthesis.
PhemeTrainingStage Which of Pheme’s two networks a training step updates.
PortaSpeechTrainingPhase The two training stages of PortaSpeech.
ProDiffTrainingPhase The two training stages of ProDiff.
PromptStyleFineTuning How PromptTTS fine-tunes its BERT style encoder on the style-factor classification task (Guo et al.
PromptTTSTrainingStage The network a PromptTTS training step updates.
SPEARTTSTrainingStage Which SPEAR-TTS network, and which of its training tasks, a training step runs.
SeedTTSTrainingComponent The independently trainable part of the Seed-TTS pipeline.
SoundStormTrainingStage Which of SoundStorm’s two networks a training step updates.
SparkTTSControlLevel The five ordinal pitch and speed labels in the released Spark tokenizer.
SparkTTSTrainingComponent The independently trainable stages of Spark-TTS.
SpeechGptTrainingStage SpeechGPT’s three training stages (§4.2).
SpeechGptVocoder The unit vocoder that voices SpeechGPT’s units (Zhang et al.
SpeechT5Stage Which of SpeechT5’s objectives SpeechT5 trains.
SpeechT5StopCriterion When SpeechT5’s generation stops.
SpiritLmModality The output modality of a SpiritLM generation (the release’s OutputModality).
SpiritLmPitchExtractor The F0 extractor of SpiritLM Expressive’s pitch tokens (§3.2).
SpiritLmStyleTokenizer Where SpiritLM Expressive’s style tokens come from.
SpiritLmUnitQuantizer How HuBERT frames become units.
SpiritLmVariant SpiritLM’s two versions (Nguyen et al.
StepAudioTrainingStage Step-Audio’s training stages (§4.2 and §5).
StyleTts2DecoderType Which waveform generator StyleTTS 2’s decoder ends in (the released configurations use iSTFTNet for LJSpeech and LibriTTS; HiFi-GAN is the code’s other option).
StyleTts2TrainingStage The modules a StyleTTS 2 training step updates (Li et al.
StyleTtsObjective Which loss weighting StyleTTS trains with.
StyleTtsTrainingStage The modules a StyleTTS training step updates (Li et al.
StyleTtsZsTrainingStage The modules a StyleTTS-ZS training step updates (Li et al.
TortoiseTrainingComponent The independently trained TorToiSe component selected by TrainingComponent.
TtsSupervision Training supervision a text-to-speech paper takes from outside the recording, which a token/mel pair does not carry and the model cannot derive itself.
UniAudioTrainingComponent Which independently trained part of UniAudio a sample updates.
VallELanguagePlacement Where VALL-E X adds its language embedding.
VallETrainingStage Which of a VALL-E-family model’s two codec language models a training step updates.
VallEXLanguage The languages of VALL-E X’s language-ID table.
VevoStage Which of Vevo’s networks Vevo trains.
WhisperSpeechTrainingStage What a WhisperSpeech training step trains.
ZonosBackbone The backbone a Zonos model uses.
ZonosConditionerKind The kinds of Zonos’s conditioners (zonos/conditioning.py).

Structs (3)

Type Summary
MoshiTimedWord A word Moshi wrote or recognized and the time in the audio it belongs to.
MoshiWord A word of a transcript with its start and end in seconds (Whisper’s word timestamps, §3.4.4).
SpiritLmWord A word of an aligned transcript: its text and its start and end in seconds.

Options & Configuration (140)

Type Summary
APNet2Options Options for APNet2 (Du et al.
APNetOptions Options for APNet (Ai and Ling 2023): an all-frame-level vocoder that predicts the log amplitude and the phase spectra directly and reconstructs the waveform by an inverse STFT.
AcousticModelOptions Base configuration options for classic acoustic TTS models.
AdaSpeech2Options Options for AdaSpeech 2 (adaptive TTS with untranscribed speech data).
AdaSpeechOptions Options for AdaSpeech (adaptive TTS with acoustic condition modeling and conditional layer normalization).
AlignTTSOptions Options for AlignTTS (feed-forward TTS that learns its own alignment with a mix density network).
AmazonPollyOptions Options for the Amazon Polly client (SynthesizeSpeech, POST /v1/speech).
AmplitudePhaseVocoderOptions Options APNet and APNet2 share (Ai and Ling 2023; Du et al.
AudioPaLMOptions Options for AudioPaLM.
AzureNeuralTTSOptions Options for the Azure AI Speech text-to-speech client (POST /cognitiveservices/v1).
BarkGenerationOptions Stage-specific sampling and generation controls.
BarkOptions Options for the Suno Bark text-prompted generative audio architecture.
BarkStageOptions Describes one of Bark’s transformer stages.
BigVGANOptions Options for BigVGAN (Lee et al.
CSMOptions Options for CSM.
ChatTTSOptions Options for ChatTTS (2noise/ChatTTS, 2024), as its released configuration (ChatTTS/config/config.py) and inference defaults (RefineTextParams, InferCodeParams) set them.
ChatterboxOptions Options for Resemble AI’s Chatterbox releases (the original English model by default; see Multilingual).
CloudTtsOptions Options every hosted text-to-speech client shares: credentials, the endpoint, the voice, the requested sample rate, retries and the time limit of a request.
CoMoSpeechOptions Options for CoMoSpeech (Ye et al.
CodecTtsOptions Base configuration options for codec-based TTS models.
CosyVoice2Options Options for CosyVoice 2.
CosyVoice3Options Options for the Fun-CosyVoice3-0.5B-2512 release.
CosyVoiceAutoregressiveOptions Shared settings for the standard CosyVoice language models and their repetition-aware sampler.
CosyVoiceCloneOptions CosyVoice release options plus the convenience voice-cloning API’s reference-duration contract.
CosyVoiceConformerOptions Native Conformer/U-Net flow and S3 encoder settings used by CosyVoice v1/v2 and Chatterbox.
CosyVoiceDiffROOptions Explicit numerical choices for DiffRO; the paper does not publish their fitted values.
CosyVoiceFamilyOptions Options shared by the CosyVoice models: audio and codec geometry, language-model dimensions, HiFT, CAM++ conditioning, optional released frontend paths and training.
CosyVoiceOptions Options for CosyVoice.
DeepVoice3Options Options for Deep Voice 3 (fully convolutional attention-based TTS), single-speaker configuration.
DiTToCodecTrainingOptions Separate Mel-VAE training, CLaM-TTS equations 4–6 and DiTTo equation 2.
DiTToTTSOptions Options for DiTTo-TTS.
DiaOptions Options for Dia.
DiffWaveOptions Options for DiffWave (Kong et al.
E2TTSOptions Options for E2 TTS (Eskimez et al.
E3TTSOptions Options for Gao et al.’s raw-waveform E3 TTS diffusion model.
ElevenLabsTTSOptions Options for the ElevenLabs text-to-speech client (POST /v1/text-to-speech/{voice_id}).
EmotiVoiceOptions Options for EmotiVoice.
EndToEndTtsOptions Base options for end-to-end TTS models that generate waveforms directly from text.
F5TTSOptions Options for F5-TTS (Chen et al.
FastSpeech2Options Options for FastSpeech 2 (variance adaptor with pitch, energy, and duration predictors).
FastSpeechOptions Options for FastSpeech (non-autoregressive TTS with duration predictor).
FireRedTTSOptions Options for FireRedTTS: the release’s sizes (FireRedTeam/FireRedTTS config_24k.json), decoding as its synthesize runs it, and the training recipes.
FishSpeechFamilyOptions Options shared by the Fish Speech models: the Dual-AR transformer, the Firefly-GAN codec, the text front end, decoding as the release’s API runs it, and training.
FishSpeechOptions Options for FishSpeech (Fish Speech 1.4).
FishSpeechV15Options Options for FishSpeechV15.
ForwardTacotronOptions Options for ForwardTacotron (non-autoregressive Tacotron with duration, pitch and energy predictors).
FreGradOptions Options for FreGrad (Nguyen et al.
GLM4VoiceOptions Options for GLM4Voice.
GPTSoVITSOptions Options for GPTSoVITS.
GlowTTSOptions Options for Glow-TTS (flow-based TTS with monotonic alignment search).
GoogleCloudTTSOptions Options for the Google Cloud Text-to-Speech client (POST /v1/text:synthesize).
GradTTSOptions Options for Grad-TTS (score-based diffusion decoder on a Glow-TTS-style text encoder).
HiFiGANOptions Options for HiFi-GAN (Kong et al.
ISTFTNetOptions Options for iSTFTNet (Kaneko et al.
IndexTTS2Options Options for IndexTTS2.
IndexTTSOptions Options for IndexTTS.
KaniTTS2Options Options for KaniTTS2.
KaniTTSOptions Options for KaniTTS.
KokoroOptions Options for Kokoro (lightweight StyleTTS2-inspired TTS with style tokens and ISTFTNet decoder).
LlamaOmniOptions Options for LlamaOmni.
LlasaOptions Options for Llasa.
MARS5TTSOptions Options for MARS5TTS.
MaskGCTOptions Options for MaskGCT.
MatchaTTSOptions Options for Matcha-TTS (Mehta et al.
MegaTTS2Options Options for MegaTTS2.
MegaTTS3Options Options for MegaTTS3: the release’s five networks (the diffusion transformer, the duration language model, the Whisper aligner, the Qwen2 grapheme-to-phoneme model and the WaveVAE), its inference settings and the paper’s training recipe.
MegaTTSOptions Options for MegaTTS.
MelGANOptions Options for MelGAN (Kumar et al.
MeloTTSOptions Options for MeloTTS.
MetaVoice1BOptions Options for the MetaVoice-1B multi-stage voice-cloning TTS model.
MinMoOptions Options for MinMo.
MoshiOptions Options for Moshi and its speech recognizer.
MultiBandMelGANOptions Options for Multi-band MelGAN (Yang et al.
MurfOptions Options for the Murf text-to-speech client (POST /v1/speech/generate).
NVIDIARivaTTSOptions Options for the NVIDIA Riva / Speech NIM text-to-speech client (POST /v1/audio/synthesize).
NaturalSpeech2Options Options for NaturalSpeech2.
NaturalSpeech3Options Options for NaturalSpeech3.
NaturalSpeechOptions Options for NaturalSpeech.
NonAttentiveTacotronOptions Options for Non-Attentive Tacotron (supervised-duration variant).
OpenVoiceOptions Options for OpenVoice TTS model.
OpenVoiceTrainingOptions Paper section 2.3 training graph.
OpenVoiceV2Options Options for OpenVoiceV2 TTS model.
OrpheusTTSOptions Options for OrpheusTTS.
OuteTTSOptions Options for OuteTTS.
ParallelWaveGANOptions Options for Parallel WaveGAN (Yamamoto et al.
ParlerTTSOptions Options for Parler-TTS (Lyth & King 2024, “Natural language guidance of high-fidelity text-to-speech with synthetic annotations”).
PhemeOptions Options for Pheme (Budzianowski et al.
PiperOptions Options for Piper (Rhasspy): VITS trained with Piper’s recipe, at one of its three model sizes.
PortaSpeechOptions Options for PortaSpeech (Ren et al.
PriorGradOptions Options for the PriorGrad vocoder (Lee et al.
ProDiffOptions Options for ProDiff (Huang et al.
PromptTTSOptions Options for PromptTTS (Guo et al.
SPEARTTSOptions Options for SPEAR-TTS (Kharitonov et al.
SeedTTSCloneOptions Options for the voice-cloning facade over Seed-TTS.
SeedTTSOptions Options for the disclosed Seed-TTS autoregressive pipeline.
SoundStormOptions Options for SoundStorm (Borsos et al.
SparkTTSOptions Options for the released Spark-TTS 0.5B graph.
SpeechGPTOptions Options for SpeechGPT and its speech recognizer.
SpeechT5Options Options for SpeechT5.
SpeedySpeechOptions Options for SpeedySpeech’s student network (fully convolutional non-autoregressive TTS).
SpiritLMOptions Options for SpiritLM.
StepAudioOptions Options for StepAudio.
StyleTTS2Options Options for StyleTTS2.
StyleTTSOptions Options for StyleTTS (Li et al.
StyleTTSZSOptions Options for StyleTTSZS.
StyleTtsBaseOptions Shared StyleTTS, StyleTTS 2 and StyleTTS-ZS audio features, style-vector width, decoder residual widths, text aligner, pitch extractor, and optimizer settings.
StyleTtsWaveformOptions The options StyleTTS 2 (Li et al.
Tacotron2Options Options for Tacotron 2 (location-sensitive attention with WaveNet vocoder).
TacotronOptions Options for Tacotron (attention-based sequence-to-sequence TTS with CBHG modules).
TortoiseTTSOptions Options for TorToiSe: Better Speech Synthesis Through Scaling (Betker, 2023).
TransformerTTSOptions Options for Transformer TTS (Li et al.
TtsModelOptions Base configuration options for text-to-speech models.
UniAudioCodecTrainingOptions The released universal codec’s adversarial-pretraining configuration.
UniAudioOptions Options for UniAudio’s universal codec and multi-scale Transformer.
UnivNetOptions Options for UnivNet (Jang et al.
VALLE2Options Options for VALL-E 2 (Chen et al.
VALLEOptions Options for VALL-E (Wang et al.
VALLEXCloneOptions Voice-cloning facade options for the native VALL-E X architecture.
VALLEXOptions Options for VALL-E X (Zhang et al.
VITS2Options Options for VITS2 (Kong et al.
VITSOptions Options for VITS (Kim et al.
VallEModelOptions The native codec, Transformer and decoding configuration shared by the VALL-E family.
VevoOptions Options for Vevo.
VitsModelOptions Options shared by the VITS family (VITS, VITS2 and the models built on them): the text encoder, the posterior encoder, the flow, the HiFi-GAN decoder and discriminators, the training objective and the optimizer.
VocoderOptions Base configuration options for neural vocoder models.
VocosOptions Options for Vocos (Siuzdak 2024): a ConvNeXt backbone at frame rate whose head predicts STFT magnitude and phase, inverted by the iSTFT, trained against multi-period and multi-resolution discriminators.
VoiceCloningOptions Base options for voice cloning TTS models.
VoiceCraftOptions Configuration for VoiceCraft (Peng et al.
VoiceFlowOptions Options for VoiceFlow (Guo et al.
VoiceboxOptions Configuration of the English Voicebox audio and duration models.
WaveGlowOptions Options for WaveGlow (Prenger et al.
WaveGradOptions Options for WaveGrad (Chen et al.
WaveNetOptions Options for the WaveNet vocoder (van den Oord et al.
WaveRNNOptions Options for WaveRNN (Kalchbrenner et al.
WellSaidLabsOptions Options for the WellSaid Labs text-to-speech client (POST /v1/tts/stream).
WhisperSpeechOptions Options for WhisperSpeech.
XTTSv2CloneOptions Options for the voice-cloning facade over the canonical XTTS-v2 graph.
XTTSv2Options Configuration of Casanova et al.’s XTTS-v2 and Coqui’s released checkpoint.
YourTTSOptions Options for YourTTS (Casanova et al.
ZonosOptions Options for Zonos.

Helpers & Utilities (3)

Type Summary
LengthRegulator FastSpeech’s length regulator (Ren et al.
MeloSymbols MeloTTS’s symbol set and language tables (myshell-ai/MeloTTS melo/text/symbols.py): 219 symbols shared by its languages, 16 tones (each language’s tones offset by its start) and 8 language ids; and its add_blank.
MonotonicAlignment Monotonic text-to-speech alignment over a token-by-frame score matrix: every frame belongs to one token, the first frame to the first token and the last to the last, and the token index stays or advances by one from frame to frame.

Exceptions (1)

Type Summary
CloudTtsException The failure of a hosted text-to-speech request: the service, its HTTP status and the start of its error body.