ChatTTS<T>

Models & Types · AiDotNet.TextToSpeech.CodecBased

ChatTTS (2noise, 2024): conversational text-to-speech as a GPT over text and DVAE audio codes, with prosody control tokens, a speaker embedding, and Vocos rendering.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → ChatTTS<T>

Implements: ICodecTts<T>

For Beginners

ChatTTS reads text aloud the way people talk, with pauses and laughs, in a random voice or one you give it.

How It Works

GPT: a Llama decoder (LlamaNetwork without its own token table or head) reads one embedding per position: a text id’s from the text table, or an audio frame’s as the sum of its 4 codes’ embeddings. Two kinds of weight-normalized heads read the last hidden state: the text head (refining text) and 4 code heads (one per code of the next frame).

Synthesis (Chat.infer): the text is first refined ([Sbreak] text [Pbreak], sampled with the text head at temperature 0.7, top-p 0.7, top-k 20, until [Ebreak]), which adds [uv_break], [laugh] and similar tokens; then [Stts], [spk_emb] (or [empty_spk]), the text and [Ptts] prompt code generation: 4 codes per frame sampled at temperature 0.3 after a repetition penalty of 1.05 over the last 16 frames, top-p 0.7 and top-k 20 (each keeping at least 3), until any code is the end id (625). The speaker embedding, L2-normalized, replaces the [spk_emb] position’s embedding; a recording’s DVAE codes can follow the text as audio positions. The GPT’s hidden states that produced the frames go through the hidden-state “Decoder” to 100-band mels (or the codes through the DVAE), and Vocos renders 24 kHz audio.

Training: 2noise released no training code. Train trains the GPT, its embeddings and heads on the code-generation sequence: the decorated text, then the utterance’s DVAE codes and the end frame, with each frame’s 4 codes predicted from the hidden state before it (the mean cross-entropy of the 4 heads). The DVAE, the Decoder and Vocos stay as loaded.

Text is bert-base-chinese’s WordPiece (shipped with the library) plus ChatTTS’s 50 control tokens, as the released tokenizer is. Load the released networks with LoadPretrainedWeights.

Example

var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional,
    NeuralNetworkTaskType.Generative, inputSize: 1, outputSize: 1);
var model = new ChatTTS<float>(architecture, new ChatTTSOptions());
model.LoadPretrainedWeights("ChatTTS/asset");
Tensor<float> audio = model.Synthesize("Hello there [uv_break] how are you?");

Constructors

Constructor Summary
ChatTTS(NeuralNetworkArchitecture<T> architecture, ChatTTSOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a native, trainable ChatTTS.
ChatTTS(NeuralNetworkArchitecture<T> architecture, string modelPath, ChatTTSOptions options) Creates a native, trainable ChatTTS.

Properties

Property Summary
int CodecFrameRate { get; }
int CodecTokenCodebooks { get; }
int CodecTokenVocabulary { get; }
int EndCode { get; } The end id of the code heads (625: one past the DVAE’s codes).
int MaxTextLength { get; }
int NumCodebooks { get; }
int TextVocabulary { get; } The text vocabulary: bert-base-chinese’s 21,128 and the 50 control tokens.

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> DecodeFromTokens(Tensor<T> tokens) The waveform of code frames [frames, NumVq] (DVAE decoder, then Vocos).
Tensor<T> EncodeToTokens(Tensor<T> audio) The DVAE codes [frames, NumVq] of a recording.
Tensor<T> SampleRandomSpeaker(Random random) A random speaker embedding (Speaker.sample_random): N(0, 1) per dimension, scaled by the speaker statistics’ std and shifted by their mean once LoadSpeakerStatistics has loaded them.
TtsVoice<T> CreateVoice(Tensor<T> speakerEmbedding, Tensor<T> recording, string transcript) A voice for Voice: a speaker embedding ([HiddenDim], e.g.
int[] Tokenize(string text) The token ids of text: ChatTTS’s control tokens as themselves, everything else through bert-base-chinese’s WordPiece (each CJK character a word, lowercased), as its BertTokenizerFast with added tokens does.
int[][] EncodeCodes(Tensor<T> audio) The DVAE codes [frames][NumVq] of a 24 kHz recording (DVAE.sample_audio).
void LoadPretrainedWeights(string assetDirectory) Loads the released networks from ChatTTS’s asset directory: gpt/model.safetensors (the LlamaModel), Embed.safetensors (tables and weight-normalized heads), DVAE.safetensors, Decoder.safetensors and Vocos.safetensors.
void LoadSpeakerStatistics(double[] statistics) Sets the speaker statistics (spk_stat: the std, then the mean, [2 · HiddenDim]).