Text To Speech
418 public types in this namespace, organized by kind.
Models & Types (151)
| Type | Summary |
|---|---|
APNet2<T> |
APNet2: a high-quality, high-efficiency vocoder that predicts the amplitude and phase spectra directly with ConvNeXt v2 networks and reconstructs the waveform by an inverse STFT, trained with linear anti-wrapping phase losses and a hinge GA… |
APNet<T> |
APNet: an all-frame-level neural vocoder that predicts the log amplitude spectrum (ASP) and the phase spectrum (PSP) directly from the mel spectrogram with residual convolution networks and reconstructs the waveform by an inverse STFT. |
AcousticConditioning<T> |
The phoneme hidden sequence after a variance-adaptor model adds its acoustic conditions, with what its decoder is conditioned on and any auxiliary loss the conditioning contributes in training. |
AcousticTargets<T> |
An utterance’s acoustic training targets, aligned to its mel frames. |
AdaSpeech2<T> |
AdaSpeech 2: adaptive TTS that adapts to a new voice from untranscribed speech, through a mel-spectrogram encoder aligned to the phoneme encoder’s output space. |
AdaSpeech<T> |
AdaSpeech: adaptive TTS for custom voice with acoustic condition modeling and conditional layer normalization. |
AlignTTS<T> |
AlignTTS: alignment-free non-autoregressive TTS using a feed-forward Transformer and mix density network alignment. |
AmazonPolly<T> |
A client of Amazon Polly’s SynthesizeSpeech API. |
AudioPaLM<T> |
AudioPaLM (Rubenstein et al. |
AzureNeuralTTS<T> |
A client of the Azure AI Speech text-to-speech REST API. |
BarkGenerationResult<T> |
Structured Bark output, including every intermediate representation. |
BarkHistoryPrompt |
Optional semantic and acoustic history used to preserve a Bark voice/prompt. |
BarkModel<T> |
Low-level, checkpoint-faithful Bark foundation model. |
Bark<T> |
Beginner-friendly text façade over the shared BarkModel foundation model. |
BigVGAN<T> |
BigVGAN: a universal GAN vocoder — HiFi-GAN with anti-aliased periodic (Snake) activations, trained at scale against multi-period and multi-resolution discriminators. |
CSM<T> |
CSM, Sesame’s Conversational Speech Model (Sesame 2025, “Crossing the uncanny valley of conversational voice”): a Llama backbone reads the conversation — text and Mimi’s audio codes interleaved, one frame per position — and predicts each ne… |
ChatTTS<T> |
ChatTTS (2noise, 2024): conversational text-to-speech as a GPT over text and DVAE audio codes, with prosody control tokens, a speaker embedding, and Vocos rendering. |
Chatterbox<T> |
Resemble AI’s Chatterbox: an emotion-controllable zero-shot T3 model whose speech tokens are rendered by S3Gen. |
CoMoSpeech<T> |
CoMoSpeech: one-step speech synthesis with a consistency model distilled from an EDM diffusion teacher. |
CosyVoice2<T> |
CosyVoice 2: streaming TTS with a pretrained LLM, finite-scalar-quantized speech tokens and chunk-aware causal flow matching. |
CosyVoice3<T> |
Fun-CosyVoice 3: Qwen2 speech-token generation, S3-v3 prompt tokens, causal DiT flow matching and CausalHiFT. |
CosyVoiceClone<T> |
Voice-cloning facade over the released CosyVoice supervised-token, conditional-flow-matching, and HiFT pipeline. |
CosyVoiceToken2TextReward<T> |
A trainable token-to-text sequence model that supplies a differentiable ASR reward for DiffRO. |
CosyVoice<T> |
CosyVoice: multilingual TTS with supervised semantic tokens and conditional flow matching. |
CsmSegment<T> |
A turn of conversation CSM conditions on: who spoke (Speaker), what they said (Text), and the recording of it (Audio, at the model’s 24 kHz). |
DeepVoice3<T> |
Deep Voice 3: fully convolutional attention-based text-to-speech. |
DiTToCheckpointPaths |
Files used to assemble DiTTo-TTS. |
DiTToTTS<T> |
DiTTo-TTS: scalable zero-shot text-to-speech with a diffusion Transformer in CLaM-TTS Mel-VAE space. |
Dia<T> |
Dia, Nari Labs’ dialogue model (2025): a Transformer encoder reads the script’s UTF-8 bytes — speaker tags [S1] and [S2] as bytes 1 and 2, non-verbal cues such as “(laughs)” as plain text — and an encoder-decoder Transformer predicts DA… |
DiffWave<T> |
DiffWave: a versatile diffusion model for audio synthesis — a non-autoregressive noise predictor built from bidirectional dilated convolutions, conditioned on the mel spectrogram, sampled by the reverse diffusion process. |
E2TTS<T> |
E2 TTS (“Embarrassingly Easy”): fully non-autoregressive text-to-speech by flow-matching speech infilling over characters padded with filler tokens to the speech length, with a flat U-Net Transformer. |
E3TTSCheckpointPaths |
Checkpoint inputs for E3’s public BERT dependency and AiDotNet-converted private U-Net weights. |
E3TTS<T> |
E3 TTS: BERT-conditioned, raw-waveform diffusion without acoustic intermediates or explicit alignment. |
ElevenLabsTTS<T> |
A client of the ElevenLabs text-to-speech API. |
EmotiVoiceFrontEnd |
EmotiVoice’s bilingual text front end (netease-youdao/EmotiVoice frontend.py, frontend_cn.py, frontend_en.py): Chinese and English text to the released model’s phoneme tokens. |
EmotiVoice<T> |
EmotiVoice: NetEase Youdao’s multi-voice, prompt-controlled emotional text-to-speech (2023). |
EnglishG2P |
An English grapheme-to-phoneme front end that writes text as espeak-ng’s American English phonemes, in the symbol sequence phonemizer produces for the TTS models trained on them (Pheme, VALL-E, and others built on USLM’s symbol table). |
F5TTS<T> |
F5-TTS: a fully non-autoregressive text-to-speech model that fills in speech with flow matching, using a Diffusion Transformer and ConvNeXt V2 text refinement over characters padded to the speech length. |
FastSpeech2<T> |
FastSpeech 2: non-autoregressive TTS with a variance adaptor for duration, pitch and energy. |
FastSpeech<T> |
FastSpeech: non-autoregressive TTS with knowledge-distilled duration predictor for parallel generation. |
FireRedTTS<T> |
FireRedTTS (Guo et al. |
FishSpeechV15<T> |
Fish Speech 1.5 (Liao et al. |
FishSpeech<T> |
Fish Speech 1.4: the Dual-AR transformer and Firefly-GAN of FishSpeechV15 (Liao et al. |
ForwardTacotron<T> |
ForwardTacotron: a non-autoregressive Tacotron that predicts phoneme durations, pitch and energy and generates the whole mel spectrogram in one pass. |
FreGrad<T> |
FreGrad: a lightweight and fast frequency-aware diffusion vocoder that denoises the waveform’s two Haar wavelet sub-bands with frequency-aware dilated convolutions. |
GLM4Voice<T> |
GLM-4-Voice (Zeng et al. |
GPTSoVITS<T> |
GPT-SoVITS (RVC-Boss, 2024): few-shot voice cloning in which a GPT writes 25 Hz semantic tokens for the text, continuing a reference recording’s, and a synthesizer speaks them in the reference voice: SoVITS’s VITS-style GAN for v1 to v2ProP… |
GlmVoiceDialogueTurn |
One conversational turn for GLM-4-Voice’s supervised fine-tuning (§4.2): the user’s speech or text, and the reply’s text and speech tokens. |
GlmVoiceReply<T> |
A GLM-4-Voice reply: its text, its speech tokens and its 22.05 kHz audio. |
GlowTTS<T> |
Glow-TTS: a flow-based generative text-to-speech model that finds its own alignment with monotonic alignment search. |
GoogleCloudTTS<T> |
A client of the Google Cloud Text-to-Speech API. |
GptSoVitsCheckpoints |
The files a GPTSoVITS loads (the release’s GPT_SoVITS/pretrained_models, Hugging Face lj1995/GPT-SoVITS). |
GradTTS<T> |
Grad-TTS: text-to-speech with a score-based diffusion decoder that turns an aligned Gaussian prior into a mel spectrogram. |
HiFiGAN<T> |
HiFi-GAN: a GAN vocoder whose generator upsamples a mel spectrogram to a waveform through transposed convolutions and multi-receptive-field residual blocks, trained against multi-period and multi-scale discriminators. |
ISTFTNet<T> |
iSTFTNet: a fast, lightweight mel-spectrogram vocoder — HiFi-GAN with its output-side upsampling replaced by an inverse STFT of a small magnitude and phase spectrogram it predicts. |
IndexTTS2<T> |
IndexTTS2 (Zhou et al. |
IndexTTS<T> |
IndexTTS (Deng et al. |
IndexTts2Checkpoints |
The files IndexTTS2’s release reads (infer_v2.py and ensure_models_available). |
KaniTTS2<T> |
KaniTTS 2 (nineninesix.ai, 2026): Kani’s LFM2 language model writing NVIDIA NanoCodec codes, with frame-level rotary positions, a learnable rotary scale per attention layer and a speaker embedding from a recording (WavLM-Large speaker embed… |
KaniTTS<T> |
KaniTTS (nineninesix.ai, 2025): text-to-speech as language modeling — an LFM2 language model reads the text and writes NVIDIA NanoCodec codes, four per frame at 12.5 frames a second, which NanoCodec’s causal decoder turns into 22.05 kHz aud… |
Kokoro<T> |
Kokoro: lightweight end-to-end TTS with a StyleTTS2-inspired architecture using style tokens and ISTFTNet decoder. |
LlamaOmniResponse<T> |
A LLaMA-Omni response: its text, its speech units and its 16 kHz audio. |
LlamaOmni<T> |
LLaMA-Omni (Fang et al., ICLR 2025): low-latency speech interaction with an LLM. |
Llasa<T> |
Llasa (Ye et al. |
MARS5TTS<T> |
MARS5 (Camb.ai, 2024): two-stage voice cloning over EnCodec’s 6 kbps codes. |
MandarinG2P |
Converts Mandarin Chinese text to IPA phonemes, as VALL-E X’s reference front end does (Plachtaa/VALL-E-X, utils/g2p/mandarin.py, chinese_to_ipa). |
Mars5Checkpoints |
The files a MARS5TTS loads. |
MaskGCT<T> |
MaskGCT (Wang et al. |
MatchaTTS<T> |
Matcha-TTS: a non-autoregressive acoustic model whose decoder is trained with optimal-transport conditional flow matching and samples a mel spectrogram in a few ODE steps. |
MegaTTS2<T> |
Mega-TTS 2 (Jiang et al. |
MegaTTS3<T> |
MegaTTS 3 (Jiang et al. |
MegaTTS<T> |
Mega-TTS (Jiang et al. |
MelGAN<T> |
MelGAN: a non-autoregressive, fully convolutional GAN vocoder — transposed-convolution upsampling with dilated residual stacks — trained against multi-scale window-based discriminators with the hinge loss and feature matching. |
MeloTTS<T> |
MeloTTS (MyShell, myshell-ai/MeloTTS): a multilingual Bert-VITS2 — VITS with VITS2’s speaker-conditioned text encoder, transformer coupling flows, noise-scaled alignment search and duration discriminator, conditioned on per- symbol tones, l… |
MetaVoice1B<T> |
MetaVoice’s learned-position causal GPT, noncausal codebook GPT, speaker LSTM and EnCodec-conditioned four-band diffusion waveform generator. |
MinMo<T> |
MinMo (Chen et al. |
MoshiConversation<T> |
A training example for Moshi’s language model: Moshi’s (the main speaker’s) 24 kHz speech, the user’s if there is one, and the main speaker’s words with their times. |
Moshi<T> |
Moshi (Défossez et al. |
MultiBandMelGAN<T> |
Multi-band MelGAN: a MelGAN generator that predicts PQMF sub-bands of the waveform at a quarter of its rate, trained with full- and sub-band multi-resolution STFT losses and multi-scale discriminators. |
Murf<T> |
A client of the Murf text-to-speech API. |
NVIDIARivaTTS<T> |
A client of an NVIDIA Riva / Speech NIM text-to-speech server’s HTTP API. |
NaturalSpeech2<T> |
NaturalSpeech 2 (Shen et al. |
NaturalSpeech3<T> |
NaturalSpeech 3 (Ju et al. |
NaturalSpeechSupPhonemes |
Sup-phonemes (Mixed-Phoneme BERT, Zhang et al. |
NaturalSpeech<T> |
NaturalSpeech (Tan et al. |
NonAttentiveTacotron<T> |
Non-Attentive Tacotron: Tacotron 2 with its attention replaced by an explicit duration predictor and Gaussian upsampling (supervised-duration variant). |
OpenVoiceV2<T> |
OpenVoiceV2: OpenVoice V2: Improved Instant Voice Cloning. |
OpenVoice<T> |
OpenVoice: versatile instant voice cloning with decoupled tone color conversion. |
OrpheusTTS<T> |
Orpheus (Canopy Labs, 2025): text-to-speech as plain language modeling — Llama 3.2 3B reads “{voice}: {text}” and continues it with SNAC codes, seven tokens per frame, which SNAC’s decoder turns into 24 kHz audio. |
OuteTTSFeatures |
OuteTTS’s audio features of a stretch of speech, each scaled to 0–100. |
OuteTTSSpeaker |
An OuteTTS 1.0 speaker profile (the release’s speaker JSON): the transcript of a short recording, its words with their codes and features, and the whole recording’s features. |
OuteTTSWord |
A word of an OuteTTS speaker profile: its text, its duration, its two DAC codebooks’ codes and its features (each 0–100). |
OuteTTS<T> |
OuteTTS 1.0 (OuteAI): text-to-speech as plain language modeling — a Llama 3.2 or Qwen3 language model reads the text and continues it with each word’s duration, energy, spectral-centroid and pitch tokens and its DAC codes, which a DAC decod… |
ParallelWaveGAN<T> |
Parallel WaveGAN: a non-autoregressive WaveNet that turns Gaussian noise into speech conditioned on a mel spectrogram, trained jointly with a multi-resolution STFT loss and an adversarial loss. |
ParlerTTS<T> |
Parler-TTS: speech in the voice a plain-language description gives (Lyth & King 2024), as the open reproduction huggingface/parler-tts builds it. |
Pheme<T> |
Pheme: efficient and conversational speech generation in two stages over SpeechTokenizer codes, a T5 text-to-semantic model and a non-autoregressive SoundStorm-style semantic-to-acoustic model. |
Piper<T> |
Piper: a fast, local neural text-to-speech system — VITS trained with Rhasspy’s recipe and model sizes. |
PortaSpeech<T> |
PortaSpeech: a portable text-to-speech model with a mixture-alignment linguistic encoder, a VAE variational generator with a normalizing-flow prior, and a flow-based post-net with grouped parameter sharing. |
PriorGrad<T> |
PriorGrad vocoder: DiffWave whose diffusion prior is a data-dependent diagonal Gaussian N(0, Σ_c) with the standard deviation of each frame taken from the mel spectrogram’s normalized frame energy. |
ProDiff<T> |
ProDiff: a progressive fast diffusion model that synthesizes a mel spectrogram in two denoising steps by predicting the clean spectrogram directly and distilling a 4-step teacher. |
PromptTTS<T> |
PromptTTS: speech whose style a plain-language prompt describes (Guo et al. |
SPEARTTS<T> |
SPEAR-TTS: multi-speaker text-to-speech trained with minimal supervision (Kharitonov et al. |
SeedTTSClone<T> |
Voice-cloning facade over Seed-TTS’s audio/text in-context continuation pipeline. |
SeedTTS<T> |
Seed-TTS’s disclosed autoregressive pipeline: speech tokenizer, causal token LM, token diffusion, and acoustic vocoder. |
SoundStorm<T> |
SoundStorm: efficient parallel audio generation (Borsos et al. |
SparkTTS<T> |
Spark-TTS: Qwen2.5 generates a single semantic speech-token stream which BiCodec renders with decoupled speaker tokens. |
SpeechGPT<T> |
SpeechGPT (Zhang et al., Findings of EMNLP 2023): a large language model with intrinsic cross-modal conversational abilities. |
SpeechGptResponse<T> |
A SpeechGPT response: its text and its units, and the speech voiced from them. |
SpeechT5Checkpoints |
The files SpeechT5’s release reads. |
SpeechT5<T> |
SpeechT5 (Ao et al. |
SpeedySpeech<T> |
SpeedySpeech: a fully convolutional non-autoregressive text-to-speech student network trained on durations from a teacher’s attention. |
SpiritLM<T> |
SpiritLM (Nguyen et al., TACL 2025): an interleaved spoken and written language model. |
SpiritLmContent<T> |
A piece of SpiritLM’s input or output: text, or 16 kHz speech. |
StepAudioSpeaker<T> |
A speaker Step-Audio’s TTS speaks as (the release’s speakers_info entry): the name, the prompt recording’s transcript and dual-codebook ids, its 22.05 kHz mel [frames, 80] and CAM++ x-vector. |
StepAudio<T> |
Step-Audio (Huang et al. |
StyleTTS2<T> |
StyleTTS 2: text-to-speech through style diffusion and adversarial training with large speech language models (Li et al. |
StyleTTSZS<T> |
StyleTTS-ZS: efficient high-quality zero-shot text-to-speech with distilled time-varying style diffusion (Li et al. |
StyleTTS<T> |
StyleTTS: a style-based generative model that speaks text in the style of a reference recording (Li et al. |
Tacotron2<T> |
Tacotron 2’s spectrogram prediction network (Shen et al. |
TacotronSpectrogram |
The mel spectrogram and frame energy that Tacotron 2 and the acoustic models following it are trained on. |
Tacotron<T> |
Tacotron: an end-to-end attention-based sequence-to-sequence text-to-speech model with CBHG modules. |
TortoiseTTS<T> |
TorToiSe: Better Speech Synthesis Through Scaling. |
TransformerTTS<T> |
Transformer TTS: Tacotron 2’s text-to-mel pipeline with its recurrent encoder and decoder replaced by a Transformer. |
TtsTrainingSample<T> |
One utterance of text-to-speech training data, with whatever supervision the model’s paper trains on. |
TtsVoice<T> |
The voice a multi-speaker or adaptive text-to-speech model synthesizes in: a speaker and, for models that read one, a reference recording of that speaker. |
UniAudio<T> |
UniAudio: a universal neural codec and multi-scale autoregressive audio Transformer. |
UnivNet<T> |
UnivNet: a GAN vocoder whose generator uses location-variable convolutions with kernels predicted from the mel spectrogram, trained against multi-resolution spectrogram and multi-period waveform discriminators. |
VALLE2<T> |
VALL-E 2: VALL-E with grouped code modeling and repetition-aware sampling, which predicts several first-codebook codes per autoregressive step and avoids the decoding loops of plain sampling. |
VALLEXClone<T> |
Voice-cloning facade over VALL-E X’s cross-lingual neural-codec language model. |
VALLEX<T> |
VALL-E X: VALL-E trained on English and Mandarin with a language ID, which speaks a language in the voice of a prompt recorded in another. |
VALLE<T> |
VALL-E: a neural codec language model for zero-shot text-to-speech, which continues a 3-second recording of an unseen speaker by predicting EnCodec codes from phonemes. |
VITS2<T> |
VITS2: single-stage text-to-speech that improves VITS with an adversarially trained stochastic duration predictor, noise-scaled monotonic alignment search, a Transformer block in the normalizing flows and a speaker-conditioned text encoder. |
VITS<T> |
VITS: end-to-end text-to-speech as a conditional VAE whose prior is a normalizing flow over a text encoding, with a stochastic duration predictor and a HiFi-GAN decoder trained adversarially. |
VevoCheckpoints |
The files Vevo’s release reads (models/vc/vevo/infer_*.py). |
Vevo<T> |
Vevo (Zhang et al., ICLR 2025): controllable zero-shot voice imitation by self-supervised disentanglement — HuBERT’s features quantized into content tokens (32 codes, timbre and style squeezed out) and content-style tokens (8,192 codes, tim… |
Vocos<T> |
Vocos: a GAN vocoder that generates Fourier spectral coefficients — magnitude and phase — at frame rate with a ConvNeXt backbone and turns them into audio with the inverse STFT. |
VoiceCraft<T> |
VoiceCraft: causal-masked, delayed-stacked neural-codec speech infilling and zero-shot TTS. |
VoiceFlow<T> |
VoiceFlow: a text-to-speech acoustic model that generates mel spectrograms with conditional flow matching and straightens its sampling trajectory with rectified flow. |
Voicebox<T> |
Voicebox: a text-guided speech infilling model with separate regression-duration and conditional-flow audio models. |
WaveGlow<T> |
WaveGlow: a flow-based generative network for speech synthesis — Glow’s invertible 1×1 convolutions and affine couplings over groups of audio samples, with WaveNet-like coupling networks conditioned on the mel spectrogram, trained only by m… |
WaveGrad<T> |
WaveGrad: estimating gradients for waveform generation — a diffusion vocoder whose noise predictor is conditioned on the continuous noise level √ᾱ, built from GAN-TTS-style upsampling blocks over the mel spectrogram modulated by FiLM from d… |
WaveNet<T> |
WaveNet: a generative model for raw audio — an autoregressive stack of dilated causal convolutions that predicts a softmax over μ-law-quantized samples, here locally conditioned on the mel spectrogram as a vocoder. |
WaveRNN<T> |
WaveRNN: efficient neural audio synthesis with a single recurrent layer whose state is split to predict the coarse and the fine byte of each 16-bit sample through a dual softmax. |
WellSaidLabs<T> |
A client of the WellSaid Labs text-to-speech API. |
WhisperSpeechConditions |
The cond variant’s conditions; null marks a condition as missing (its own learned code). |
WhisperSpeech<T> |
WhisperSpeech (Collabora 2024): text-to-speech by inverting Whisper, after SPEAR-TTS (Kharitonov et al. |
XTTSv2Clone<T> |
Voice-cloning facade over the canonical XTTS-v2 zero-shot architecture. |
XTTSv2<T> |
XTTS-v2: multilingual zero-shot TTS with a dVAE tokenizer, GPT-2 and speaker-conditioned HiFi-GAN. |
YourTTS<T> |
YourTTS: zero-shot multi-speaker text-to-speech and voice conversion — VITS conditioned on speaker embeddings from a pretrained speaker encoder, with language embeddings for multilingual training. |
ZonosConditioner |
One of Zonos’s conditioners as its release config describes it. |
Zonos<T> |
Zonos, Zyphra’s text-to-speech model (2025): a prefix of conditioning tokens — the text’s eSpeak phonemes, a speaker embedding, and targets for emotion, pitch variation, speaking rate, bandwidth and language — followed by DAC’s nine codeboo… |
Base Classes (16)
| Type | Summary |
|---|---|
AcousticModelBase<T> |
Base class for acoustic TTS models that generate mel-spectrograms from text. |
AmplitudePhaseVocoderBase<T> |
The frame-level amplitude-and-phase vocoders’ shared generator and losses (APNet, Ai and Ling 2023; APNet2, Du et al. |
CloudTtsClientBase<T> |
A client of a hosted text-to-speech service: it sends the text to the service’s REST API and decodes the audio it returns into a waveform tensor. |
CosyVoiceBase<T> |
What the CosyVoice models share (FunAudioLLM/CosyVoice, Apache-2.0): a language model speaking S3 tokens, a flow sampling an 80-band mel from them by conditional flow matching with classifier-free guidance, HiFT turning the mel into audio,… |
DiffusionVocoderBase<T> |
The denoising-diffusion training and sampling diffusion vocoders share (Ho et al. |
FishSpeechBase<T> |
The Fish Speech models (Liao et al. |
FlowMatchingTtsModelBase<T> |
The text-guided speech-infilling conditional flow matching shared by E2 TTS and F5-TTS (Chen et al. |
GanVocoderBase<T> |
The training loop GAN vocoders share: a generator that turns a mel spectrogram into a waveform, discriminators that tell real from generated audio, and alternating discriminator and generator steps on random segments. |
KaniTTSBase<T> |
What both KaniTTS releases share: an LFM2 language model writing NVIDIA NanoCodec codes after the release’s prompt, the codec, the speech extraction, the training sequence and its loss, and loading. |
SegmentVocoderBase<T> |
The training and synthesis plumbing of vocoders trained by one objective on aligned (mel spectrogram, waveform) segments — the diffusion vocoders (DiffWave, WaveGrad, PriorGrad, FreGrad), the flow WaveGlow and the autoregressive WaveNet and… |
StochasticDurationVitsBase<T> |
A VITS-family model whose durations come from VITS’s stochastic duration predictor (Kim et al. |
TtsModelBase<T> |
Base class for text-to-speech neural networks that can operate in both ONNX inference and native training modes. |
VallEModelBase<T> |
What the models of the VALL-E family share: EnCodec, the AR and NAR codec language models (VallECore), a phoneme vocabulary, training each model on its own, and prompted synthesis. |
VarianceAdaptorTtsModelBase<T> |
Base class for non-autoregressive acoustic models of the FastSpeech family: a phoneme encoder, a variance adaptor (duration and length regulation, optionally pitch and energy), and a mel decoder. |
VitsTtsModelBase<T> |
The VITS family’s shared model: a conditional VAE whose prior is a normalizing flow over a text encoding, aligned by monotonic alignment search, with a duration model and a HiFi-GAN decoder trained adversarially on random windows. |
VocoderBase<T> |
Base class for neural vocoder models that convert mel-spectrograms to audio waveforms. |
Interfaces (8)
| Type | Summary |
|---|---|
IAcousticModel<T> |
Interface for acoustic models that generate mel-spectrograms from text. |
ICodecTts<T> |
Interface for codec-based TTS models that use neural audio codecs with language model decoding. |
ICosyVoiceTokenReward<T> |
A differentiable token-domain reward for CosyVoice 3 DiffRO (paper equations 3-7). |
IEndToEndTts<T> |
Interface for end-to-end TTS models that generate waveforms directly from text without a separate vocoder stage. |
IStreamingTts<T> |
Interface for TTS models that support streaming/chunked synthesis with low latency. |
ITtsModel<T> |
Base interface for all text-to-speech models. |
IVocoder<T> |
Interface for neural vocoders that convert mel-spectrograms to audio waveforms. |
IVoiceCloner<T> |
Interface for TTS models that support zero-shot or few-shot voice cloning from reference audio. |
Enums (96)
| Type | Summary |
|---|---|
AdaSpeech2TrainingStep |
The steps of AdaSpeech 2’s adaptation pipeline (Yan et al. |
AdaSpeechTrainingPhase |
The training phases of AdaSpeech (Chen et al. |
AlignTTSTrainingPhase |
The training phases of AlignTTS (Zeng et al. |
AudioPalmDetokenizer |
How AudioPaLM turns audio tokens into speech (§3.3). |
AudioPalmSpeechTokenizer |
How AudioPaLM turns speech into audio tokens (Rubenstein et al. |
BarkGenerationStage |
Identifies a stage in Bark’s semantic-to-waveform generation pipeline. |
ChatterboxVariant |
The released Chatterbox checkpoint a model follows. |
CoMoSpeechTrainingPhase |
The two training stages of CoMoSpeech. |
CosyVoiceFlowLoss |
The loss CosyVoice’s flow matching minimizes. |
CosyVoiceTrainingStage |
Which CosyVoice network a training step trains (the recipes train llm, flow and hifigan in turn). |
DiTToModelSize |
The four diffusion-transformer sizes evaluated by DiTTo-TTS. |
DiTToTrainingStage |
Diffusion and length prediction, plus the separately pretrained and optionally semantically finetuned codec. |
DiffWaveConditioner |
What DiffWave’s noise predictor is conditioned on (Kong et al. |
FireRedTtsEmotion |
An emotion of FireRedTTS’s instruction tuning (§4.2). |
FireRedTtsMelDecoder |
Which of the paper’s Mel decoders turns semantic tokens into the mel (§3.3). |
FireRedTtsStage |
Which of FireRedTTS’s networks a training step updates. |
FireRedTtsVocoder |
Which BigVGAN-v2 vocoder turns the 16 kHz, 10 ms mel into audio. |
FishSpeechRecipe |
Which training recipe a Fish Speech model’s optimizer follows. |
GlmVoiceDecoding |
How GLM-4-Voice’s decoder turns a sequence of speech tokens into audio. |
GlmVoiceGeneration |
How GLM-4-Voice lays out its spoken reply (Zeng et al. |
GlmVoiceTrainingStage |
Which of GLM-4-Voice’s two language-model stages a training step follows (§4). |
GlmVoiceTtsFormat |
How Synthesize reads a text aloud. |
GptSoVitsStage |
Which of GPT-SoVITS’s networks GPTSoVITS trains. |
GptSoVitsTextCut |
How GPT-SoVITS cuts the text before synthesis (how_to_cut). |
GptSoVitsTextLanguage |
How GPT-SoVITS reads the text’s language (inference_webui.dict_language). |
GptSoVitsVersion |
The GPT-SoVITS release a GPTSoVITS follows (each has its own released weights). |
IndexTts2DurationConditioning |
How IndexTTS2’s text-to-semantic model receives the target length. |
IndexTts2EmotionMode |
How IndexTTS2 turns an emotion distribution into the emotion vector. |
IndexTts2Stage |
Which of IndexTTS2’s networks IndexTTS2 trains. |
LlamaOmniAlignment |
Which hidden state stands for a response token when the speech decoder trains. |
LlamaOmniTrainingStage |
Which of LLaMA-Omni’s two training stages (§2.5) a step follows. |
Mars5Stage |
Which of MARS5’s networks MARS5TTS trains. |
MaskGctActivation |
The gated activation in MaskGctTransformer’s feed-forward. |
MaskGctDurationMode |
How MaskGCT chooses the target’s length. |
MaskGctStage |
Which of MaskGCT’s networks MaskGCT trains. |
MegaTts2CodeAnchor |
Where CVQ-VAE’s dynamic initialization draws the anchors that move rarely used codes. |
MegaTts2CodeDistance |
How the CVQ-VAE quantizer of MegaTTS2 finds a vector’s code. |
MegaTts2Stage |
Which of Mega-TTS 2’s training stages MegaTTS2 runs. |
MegaTtsLanguage |
The front end MegaTTS reads text with. |
MegaTtsStage |
Which of Mega-TTS’s two models MegaTTS trains. |
MeloLanguage |
The languages MeloTTS’s checkpoints are trained for (one per checkpoint). |
MinMoDuplexAction |
What the full duplex predictor decides at a position of the conversation. |
MinMoTrainingStage |
MinMo’s training stages (§3.4). |
MinMoVoiceEncoder |
Which voice encoder MinMo reads speech with. |
MoshiDepthLayout |
Which audio streams Moshi’s Depth Transformer predicts. |
MoshiTrainingStage |
Moshi’s training stages (§4.4, Table 1). |
MoshiVariant |
The delay between Moshi’s text and audio streams (§3.4.4, App. |
NaturalSpeech2CeResidual |
The residual each quantizer’s cross-entropy term (L_ce-rvq) is measured on. |
NaturalSpeech2Diffusion |
What NaturalSpeech 2’s latent diffusion learns. |
NaturalSpeech2Positions |
The positional encoding of NaturalSpeech 2’s Transformers. |
NaturalSpeech2PromptPosition |
Where the training prompt is cut from the utterance. |
NaturalSpeech2ScoreLoss |
The form of Eq. |
NaturalSpeech2Solver |
The ODE solver of reverse diffusion. |
NaturalSpeech3Stage |
What NaturalSpeech3’s training step trains. |
NaturalSpeechPhonemeEncoder |
NaturalSpeech’s phoneme encoder. |
NaturalSpeechStage |
NaturalSpeech’s training stages (Tan et al. |
NaturalSpeechUpsampling |
How NaturalSpeech’s learnable upsampling layer is built. |
OrpheusDecoding |
How OrpheusTTS turns the generated codes into audio. |
OuteTTSBackbone |
The language model OuteTTS 1.0 is built on. |
PalmDecoderKind |
Which decoder PalmDecoder builds. |
PhemeAcousticDecoding |
How Pheme’s acoustic model decodes its codebooks at synthesis. |
PhemeTrainingStage |
Which of Pheme’s two networks a training step updates. |
PortaSpeechTrainingPhase |
The two training stages of PortaSpeech. |
ProDiffTrainingPhase |
The two training stages of ProDiff. |
PromptStyleFineTuning |
How PromptTTS fine-tunes its BERT style encoder on the style-factor classification task (Guo et al. |
PromptTTSTrainingStage |
The network a PromptTTS training step updates. |
SPEARTTSTrainingStage |
Which SPEAR-TTS network, and which of its training tasks, a training step runs. |
SeedTTSTrainingComponent |
The independently trainable part of the Seed-TTS pipeline. |
SoundStormTrainingStage |
Which of SoundStorm’s two networks a training step updates. |
SparkTTSControlLevel |
The five ordinal pitch and speed labels in the released Spark tokenizer. |
SparkTTSTrainingComponent |
The independently trainable stages of Spark-TTS. |
SpeechGptTrainingStage |
SpeechGPT’s three training stages (§4.2). |
SpeechGptVocoder |
The unit vocoder that voices SpeechGPT’s units (Zhang et al. |
SpeechT5Stage |
Which of SpeechT5’s objectives SpeechT5 trains. |
SpeechT5StopCriterion |
When SpeechT5’s generation stops. |
SpiritLmModality |
The output modality of a SpiritLM generation (the release’s OutputModality). |
SpiritLmPitchExtractor |
The F0 extractor of SpiritLM Expressive’s pitch tokens (§3.2). |
SpiritLmStyleTokenizer |
Where SpiritLM Expressive’s style tokens come from. |
SpiritLmUnitQuantizer |
How HuBERT frames become units. |
SpiritLmVariant |
SpiritLM’s two versions (Nguyen et al. |
StepAudioTrainingStage |
Step-Audio’s training stages (§4.2 and §5). |
StyleTts2DecoderType |
Which waveform generator StyleTTS 2’s decoder ends in (the released configurations use iSTFTNet for LJSpeech and LibriTTS; HiFi-GAN is the code’s other option). |
StyleTts2TrainingStage |
The modules a StyleTTS 2 training step updates (Li et al. |
StyleTtsObjective |
Which loss weighting StyleTTS trains with. |
StyleTtsTrainingStage |
The modules a StyleTTS training step updates (Li et al. |
StyleTtsZsTrainingStage |
The modules a StyleTTS-ZS training step updates (Li et al. |
TortoiseTrainingComponent |
The independently trained TorToiSe component selected by TrainingComponent. |
TtsSupervision |
Training supervision a text-to-speech paper takes from outside the recording, which a token/mel pair does not carry and the model cannot derive itself. |
UniAudioTrainingComponent |
Which independently trained part of UniAudio a sample updates. |
VallELanguagePlacement |
Where VALL-E X adds its language embedding. |
VallETrainingStage |
Which of a VALL-E-family model’s two codec language models a training step updates. |
VallEXLanguage |
The languages of VALL-E X’s language-ID table. |
VevoStage |
Which of Vevo’s networks Vevo trains. |
WhisperSpeechTrainingStage |
What a WhisperSpeech training step trains. |
ZonosBackbone |
The backbone a Zonos model uses. |
ZonosConditionerKind |
The kinds of Zonos’s conditioners (zonos/conditioning.py). |
Structs (3)
| Type | Summary |
|---|---|
MoshiTimedWord |
A word Moshi wrote or recognized and the time in the audio it belongs to. |
MoshiWord |
A word of a transcript with its start and end in seconds (Whisper’s word timestamps, §3.4.4). |
SpiritLmWord |
A word of an aligned transcript: its text and its start and end in seconds. |
Options & Configuration (140)
| Type | Summary |
|---|---|
APNet2Options |
Options for APNet2 (Du et al. |
APNetOptions |
Options for APNet (Ai and Ling 2023): an all-frame-level vocoder that predicts the log amplitude and the phase spectra directly and reconstructs the waveform by an inverse STFT. |
AcousticModelOptions |
Base configuration options for classic acoustic TTS models. |
AdaSpeech2Options |
Options for AdaSpeech 2 (adaptive TTS with untranscribed speech data). |
AdaSpeechOptions |
Options for AdaSpeech (adaptive TTS with acoustic condition modeling and conditional layer normalization). |
AlignTTSOptions |
Options for AlignTTS (feed-forward TTS that learns its own alignment with a mix density network). |
AmazonPollyOptions |
Options for the Amazon Polly client (SynthesizeSpeech, POST /v1/speech). |
AmplitudePhaseVocoderOptions |
Options APNet and APNet2 share (Ai and Ling 2023; Du et al. |
AudioPaLMOptions |
Options for AudioPaLM. |
AzureNeuralTTSOptions |
Options for the Azure AI Speech text-to-speech client (POST /cognitiveservices/v1). |
BarkGenerationOptions |
Stage-specific sampling and generation controls. |
BarkOptions |
Options for the Suno Bark text-prompted generative audio architecture. |
BarkStageOptions |
Describes one of Bark’s transformer stages. |
BigVGANOptions |
Options for BigVGAN (Lee et al. |
CSMOptions |
Options for CSM. |
ChatTTSOptions |
Options for ChatTTS (2noise/ChatTTS, 2024), as its released configuration (ChatTTS/config/config.py) and inference defaults (RefineTextParams, InferCodeParams) set them. |
ChatterboxOptions |
Options for Resemble AI’s Chatterbox releases (the original English model by default; see Multilingual). |
CloudTtsOptions |
Options every hosted text-to-speech client shares: credentials, the endpoint, the voice, the requested sample rate, retries and the time limit of a request. |
CoMoSpeechOptions |
Options for CoMoSpeech (Ye et al. |
CodecTtsOptions |
Base configuration options for codec-based TTS models. |
CosyVoice2Options |
Options for CosyVoice 2. |
CosyVoice3Options |
Options for the Fun-CosyVoice3-0.5B-2512 release. |
CosyVoiceAutoregressiveOptions |
Shared settings for the standard CosyVoice language models and their repetition-aware sampler. |
CosyVoiceCloneOptions |
CosyVoice release options plus the convenience voice-cloning API’s reference-duration contract. |
CosyVoiceConformerOptions |
Native Conformer/U-Net flow and S3 encoder settings used by CosyVoice v1/v2 and Chatterbox. |
CosyVoiceDiffROOptions |
Explicit numerical choices for DiffRO; the paper does not publish their fitted values. |
CosyVoiceFamilyOptions |
Options shared by the CosyVoice models: audio and codec geometry, language-model dimensions, HiFT, CAM++ conditioning, optional released frontend paths and training. |
CosyVoiceOptions |
Options for CosyVoice. |
DeepVoice3Options |
Options for Deep Voice 3 (fully convolutional attention-based TTS), single-speaker configuration. |
DiTToCodecTrainingOptions |
Separate Mel-VAE training, CLaM-TTS equations 4–6 and DiTTo equation 2. |
DiTToTTSOptions |
Options for DiTTo-TTS. |
DiaOptions |
Options for Dia. |
DiffWaveOptions |
Options for DiffWave (Kong et al. |
E2TTSOptions |
Options for E2 TTS (Eskimez et al. |
E3TTSOptions |
Options for Gao et al.’s raw-waveform E3 TTS diffusion model. |
ElevenLabsTTSOptions |
Options for the ElevenLabs text-to-speech client (POST /v1/text-to-speech/{voice_id}). |
EmotiVoiceOptions |
Options for EmotiVoice. |
EndToEndTtsOptions |
Base options for end-to-end TTS models that generate waveforms directly from text. |
F5TTSOptions |
Options for F5-TTS (Chen et al. |
FastSpeech2Options |
Options for FastSpeech 2 (variance adaptor with pitch, energy, and duration predictors). |
FastSpeechOptions |
Options for FastSpeech (non-autoregressive TTS with duration predictor). |
FireRedTTSOptions |
Options for FireRedTTS: the release’s sizes (FireRedTeam/FireRedTTS config_24k.json), decoding as its synthesize runs it, and the training recipes. |
FishSpeechFamilyOptions |
Options shared by the Fish Speech models: the Dual-AR transformer, the Firefly-GAN codec, the text front end, decoding as the release’s API runs it, and training. |
FishSpeechOptions |
Options for FishSpeech (Fish Speech 1.4). |
FishSpeechV15Options |
Options for FishSpeechV15. |
ForwardTacotronOptions |
Options for ForwardTacotron (non-autoregressive Tacotron with duration, pitch and energy predictors). |
FreGradOptions |
Options for FreGrad (Nguyen et al. |
GLM4VoiceOptions |
Options for GLM4Voice. |
GPTSoVITSOptions |
Options for GPTSoVITS. |
GlowTTSOptions |
Options for Glow-TTS (flow-based TTS with monotonic alignment search). |
GoogleCloudTTSOptions |
Options for the Google Cloud Text-to-Speech client (POST /v1/text:synthesize). |
GradTTSOptions |
Options for Grad-TTS (score-based diffusion decoder on a Glow-TTS-style text encoder). |
HiFiGANOptions |
Options for HiFi-GAN (Kong et al. |
ISTFTNetOptions |
Options for iSTFTNet (Kaneko et al. |
IndexTTS2Options |
Options for IndexTTS2. |
IndexTTSOptions |
Options for IndexTTS. |
KaniTTS2Options |
Options for KaniTTS2. |
KaniTTSOptions |
Options for KaniTTS. |
KokoroOptions |
Options for Kokoro (lightweight StyleTTS2-inspired TTS with style tokens and ISTFTNet decoder). |
LlamaOmniOptions |
Options for LlamaOmni. |
LlasaOptions |
Options for Llasa. |
MARS5TTSOptions |
Options for MARS5TTS. |
MaskGCTOptions |
Options for MaskGCT. |
MatchaTTSOptions |
Options for Matcha-TTS (Mehta et al. |
MegaTTS2Options |
Options for MegaTTS2. |
MegaTTS3Options |
Options for MegaTTS3: the release’s five networks (the diffusion transformer, the duration language model, the Whisper aligner, the Qwen2 grapheme-to-phoneme model and the WaveVAE), its inference settings and the paper’s training recipe. |
MegaTTSOptions |
Options for MegaTTS. |
MelGANOptions |
Options for MelGAN (Kumar et al. |
MeloTTSOptions |
Options for MeloTTS. |
MetaVoice1BOptions |
Options for the MetaVoice-1B multi-stage voice-cloning TTS model. |
MinMoOptions |
Options for MinMo. |
MoshiOptions |
Options for Moshi and its speech recognizer. |
MultiBandMelGANOptions |
Options for Multi-band MelGAN (Yang et al. |
MurfOptions |
Options for the Murf text-to-speech client (POST /v1/speech/generate). |
NVIDIARivaTTSOptions |
Options for the NVIDIA Riva / Speech NIM text-to-speech client (POST /v1/audio/synthesize). |
NaturalSpeech2Options |
Options for NaturalSpeech2. |
NaturalSpeech3Options |
Options for NaturalSpeech3. |
NaturalSpeechOptions |
Options for NaturalSpeech. |
NonAttentiveTacotronOptions |
Options for Non-Attentive Tacotron (supervised-duration variant). |
OpenVoiceOptions |
Options for OpenVoice TTS model. |
OpenVoiceTrainingOptions |
Paper section 2.3 training graph. |
OpenVoiceV2Options |
Options for OpenVoiceV2 TTS model. |
OrpheusTTSOptions |
Options for OrpheusTTS. |
OuteTTSOptions |
Options for OuteTTS. |
ParallelWaveGANOptions |
Options for Parallel WaveGAN (Yamamoto et al. |
ParlerTTSOptions |
Options for Parler-TTS (Lyth & King 2024, “Natural language guidance of high-fidelity text-to-speech with synthetic annotations”). |
PhemeOptions |
Options for Pheme (Budzianowski et al. |
PiperOptions |
Options for Piper (Rhasspy): VITS trained with Piper’s recipe, at one of its three model sizes. |
PortaSpeechOptions |
Options for PortaSpeech (Ren et al. |
PriorGradOptions |
Options for the PriorGrad vocoder (Lee et al. |
ProDiffOptions |
Options for ProDiff (Huang et al. |
PromptTTSOptions |
Options for PromptTTS (Guo et al. |
SPEARTTSOptions |
Options for SPEAR-TTS (Kharitonov et al. |
SeedTTSCloneOptions |
Options for the voice-cloning facade over Seed-TTS. |
SeedTTSOptions |
Options for the disclosed Seed-TTS autoregressive pipeline. |
SoundStormOptions |
Options for SoundStorm (Borsos et al. |
SparkTTSOptions |
Options for the released Spark-TTS 0.5B graph. |
SpeechGPTOptions |
Options for SpeechGPT and its speech recognizer. |
SpeechT5Options |
Options for SpeechT5. |
SpeedySpeechOptions |
Options for SpeedySpeech’s student network (fully convolutional non-autoregressive TTS). |
SpiritLMOptions |
Options for SpiritLM. |
StepAudioOptions |
Options for StepAudio. |
StyleTTS2Options |
Options for StyleTTS2. |
StyleTTSOptions |
Options for StyleTTS (Li et al. |
StyleTTSZSOptions |
Options for StyleTTSZS. |
StyleTtsBaseOptions |
Shared StyleTTS, StyleTTS 2 and StyleTTS-ZS audio features, style-vector width, decoder residual widths, text aligner, pitch extractor, and optimizer settings. |
StyleTtsWaveformOptions |
The options StyleTTS 2 (Li et al. |
Tacotron2Options |
Options for Tacotron 2 (location-sensitive attention with WaveNet vocoder). |
TacotronOptions |
Options for Tacotron (attention-based sequence-to-sequence TTS with CBHG modules). |
TortoiseTTSOptions |
Options for TorToiSe: Better Speech Synthesis Through Scaling (Betker, 2023). |
TransformerTTSOptions |
Options for Transformer TTS (Li et al. |
TtsModelOptions |
Base configuration options for text-to-speech models. |
UniAudioCodecTrainingOptions |
The released universal codec’s adversarial-pretraining configuration. |
UniAudioOptions |
Options for UniAudio’s universal codec and multi-scale Transformer. |
UnivNetOptions |
Options for UnivNet (Jang et al. |
VALLE2Options |
Options for VALL-E 2 (Chen et al. |
VALLEOptions |
Options for VALL-E (Wang et al. |
VALLEXCloneOptions |
Voice-cloning facade options for the native VALL-E X architecture. |
VALLEXOptions |
Options for VALL-E X (Zhang et al. |
VITS2Options |
Options for VITS2 (Kong et al. |
VITSOptions |
Options for VITS (Kim et al. |
VallEModelOptions |
The native codec, Transformer and decoding configuration shared by the VALL-E family. |
VevoOptions |
Options for Vevo. |
VitsModelOptions |
Options shared by the VITS family (VITS, VITS2 and the models built on them): the text encoder, the posterior encoder, the flow, the HiFi-GAN decoder and discriminators, the training objective and the optimizer. |
VocoderOptions |
Base configuration options for neural vocoder models. |
VocosOptions |
Options for Vocos (Siuzdak 2024): a ConvNeXt backbone at frame rate whose head predicts STFT magnitude and phase, inverted by the iSTFT, trained against multi-period and multi-resolution discriminators. |
VoiceCloningOptions |
Base options for voice cloning TTS models. |
VoiceCraftOptions |
Configuration for VoiceCraft (Peng et al. |
VoiceFlowOptions |
Options for VoiceFlow (Guo et al. |
VoiceboxOptions |
Configuration of the English Voicebox audio and duration models. |
WaveGlowOptions |
Options for WaveGlow (Prenger et al. |
WaveGradOptions |
Options for WaveGrad (Chen et al. |
WaveNetOptions |
Options for the WaveNet vocoder (van den Oord et al. |
WaveRNNOptions |
Options for WaveRNN (Kalchbrenner et al. |
WellSaidLabsOptions |
Options for the WellSaid Labs text-to-speech client (POST /v1/tts/stream). |
WhisperSpeechOptions |
Options for WhisperSpeech. |
XTTSv2CloneOptions |
Options for the voice-cloning facade over the canonical XTTS-v2 graph. |
XTTSv2Options |
Configuration of Casanova et al.’s XTTS-v2 and Coqui’s released checkpoint. |
YourTTSOptions |
Options for YourTTS (Casanova et al. |
ZonosOptions |
Options for Zonos. |
Helpers & Utilities (3)
| Type | Summary |
|---|---|
LengthRegulator |
FastSpeech’s length regulator (Ren et al. |
MeloSymbols |
MeloTTS’s symbol set and language tables (myshell-ai/MeloTTS melo/text/symbols.py): 219 symbols shared by its languages, 16 tones (each language’s tones offset by its start) and 8 language ids; and its add_blank. |
MonotonicAlignment |
Monotonic text-to-speech alignment over a token-by-frame score matrix: every frame belongs to one token, the first frame to the first token and the last to the last, and the token index stays or advances by one from frame to frame. |
Exceptions (1)
| Type | Summary |
|---|---|
CloudTtsException |
The failure of a hosted text-to-speech request: the service, its HTTP status and the start of its error body. |