ChatTTS<T>
Models & Types · AiDotNet.TextToSpeech.CodecBased
ChatTTS (2noise, 2024): conversational text-to-speech as a GPT over text and DVAE audio codes, with prosody control tokens, a speaker embedding, and Vocos rendering.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → ChatTTS<T>
Implements: ICodecTts<T>
For Beginners
ChatTTS reads text aloud the way people talk, with pauses and laughs, in a random voice or one you give it.
How It Works
GPT: a Llama decoder (LlamaNetwork without its own token table or head) reads one embedding
per position: a text id’s from the text table, or an audio frame’s as the sum of its 4 codes’ embeddings. Two kinds
of weight-normalized heads read the last hidden state: the text head (refining text) and 4 code heads (one per
code of the next frame).
Synthesis (Chat.infer): the text is first refined ([Sbreak] text [Pbreak], sampled with
the text head at temperature 0.7, top-p 0.7, top-k 20, until [Ebreak]), which adds [uv_break],
[laugh] and similar tokens; then [Stts], [spk_emb] (or [empty_spk]), the text and
[Ptts] prompt code generation: 4 codes per frame sampled at temperature 0.3 after a repetition penalty of 1.05
over the last 16 frames, top-p 0.7 and top-k 20 (each keeping at least 3), until any code is the end id (625).
The speaker embedding, L2-normalized, replaces the [spk_emb] position’s embedding; a recording’s DVAE codes
can follow the text as audio positions. The GPT’s hidden states that produced the frames go through the
hidden-state “Decoder” to 100-band mels (or the codes through the DVAE), and Vocos renders 24 kHz audio.
Training: 2noise released no training code. Train
trains the GPT, its embeddings and heads on the code-generation sequence: the decorated text, then the utterance’s
DVAE codes and the end frame, with each frame’s 4 codes predicted from the hidden state before it (the mean
cross-entropy of the 4 heads). The DVAE, the Decoder and Vocos stay as loaded.
Text is bert-base-chinese’s WordPiece (shipped with the library) plus ChatTTS’s 50 control tokens, as the
released tokenizer is. Load the released networks with LoadPretrainedWeights.
Example
var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional,
NeuralNetworkTaskType.Generative, inputSize: 1, outputSize: 1);
var model = new ChatTTS<float>(architecture, new ChatTTSOptions());
model.LoadPretrainedWeights("ChatTTS/asset");
Tensor<float> audio = model.Synthesize("Hello there [uv_break] how are you?");
Constructors
| Constructor | Summary |
|---|---|
ChatTTS(NeuralNetworkArchitecture<T> architecture, ChatTTSOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a native, trainable ChatTTS. |
ChatTTS(NeuralNetworkArchitecture<T> architecture, string modelPath, ChatTTSOptions options) |
Creates a native, trainable ChatTTS. |
Properties
| Property | Summary |
|---|---|
int CodecFrameRate { get; } |
|
int CodecTokenCodebooks { get; } |
|
int CodecTokenVocabulary { get; } |
|
int EndCode { get; } |
The end id of the code heads (625: one past the DVAE’s codes). |
int MaxTextLength { get; } |
|
int NumCodebooks { get; } |
|
int TextVocabulary { get; } |
The text vocabulary: bert-base-chinese’s 21,128 and the 50 control tokens. |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> DecodeFromTokens(Tensor<T> tokens) |
The waveform of code frames [frames, NumVq] (DVAE decoder, then Vocos). |
Tensor<T> EncodeToTokens(Tensor<T> audio) |
The DVAE codes [frames, NumVq] of a recording. |
Tensor<T> SampleRandomSpeaker(Random random) |
A random speaker embedding (Speaker.sample_random): N(0, 1) per dimension, scaled by the speaker statistics’ std and shifted by their mean once LoadSpeakerStatistics has loaded them. |
TtsVoice<T> CreateVoice(Tensor<T> speakerEmbedding, Tensor<T> recording, string transcript) |
A voice for Voice: a speaker embedding ([HiddenDim], e.g. |
int[] Tokenize(string text) |
The token ids of text: ChatTTS’s control tokens as themselves, everything else through bert-base-chinese’s WordPiece (each CJK character a word, lowercased), as its BertTokenizerFast with added tokens does. |
int[][] EncodeCodes(Tensor<T> audio) |
The DVAE codes [frames][NumVq] of a 24 kHz recording (DVAE.sample_audio). |
void LoadPretrainedWeights(string assetDirectory) |
Loads the released networks from ChatTTS’s asset directory: gpt/model.safetensors (the LlamaModel), Embed.safetensors (tables and weight-normalized heads), DVAE.safetensors, Decoder.safetensors and Vocos.safetensors. |
void LoadSpeakerStatistics(double[] statistics) |
Sets the speaker statistics (spk_stat: the std, then the mean, [2 · HiddenDim]). |