IndexTTS2<T>
Models & Types · AiDotNet.TextToSpeech.Latest
IndexTTS2 (Zhou et al. 2025): zero-shot autoregressive text-to-speech with precise duration control and emotion control separated from the speaker’s timbre — text to semantic tokens by a GPT-2 conditioned on a speaker prompt and a style prompt, semantic tokens (with the GPT’s latents) to a mel spectrogram by flow matching, and BigVGAN-v2 to audio.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → IndexTTS2<T>
For Beginners
Give IndexTTS2 a few seconds of someone speaking and it reads new text in that voice. You can also give it a second recording whose emotion to copy, a set of emotion weights, or a sentence describing the emotion, and say how many tokens (roughly how long) the speech should be.
How It Works
The text-to-semantic model reads [c + e, p, BT text ET, BA semantic]: c, 32 latents a conformer and
perceiver pool from the speaker prompt’s w2v-BERT 2.0 features; e, the emotion vector pooled the same way from a style
prompt (or mixed from an emotion table by weights, or by weights a text-to-emotion model reads off a sentence); p, the
duration token — by default the paper’s W_sem h(T) naming the number of semantic tokens to generate, or zeros to
generate freely. The semantic-to-mel module conditions seed-vc’s DiT on the semantic codec’s features plus the GPT’s
last hidden states (the paper’s GPT latent enhancement), the prompt’s mel and its CAM++ style.
The paper leaves most sizes to the release (IndexTeam/IndexTTS-2), whose code fills the gaps; where the
two differ the paper is the default and the release an option (Release), which the
released weights need.
Example
var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var options = IndexTTS2Options.Release();
options.BpeModelPath = "IndexTTS-2/bpe.model";
var model = new IndexTTS2<float>(architecture, options);
model.LoadReferenceWeights(new IndexTts2Checkpoints { Directory = "IndexTTS-2", SemanticModel = "w2v-bert-2.0/model.safetensors",
SemanticCodec = "MaskGCT/semantic_codec/model.safetensors", SpeakerEncoder = "campplus_cn_common.bin",
Vocoder = "bigvgan_v2_22khz_80band_256x/bigvgan_generator.pt" });
model.Voice = model.CreateVoice(promptAudio, promptSampleRate);
var audio = model.Synthesize("Text to speak in that voice."); // 22.05 kHz
Constructors
| Constructor | Summary |
|---|---|
IndexTTS2(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
IndexTTS2(NeuralNetworkArchitecture<T> architecture, IndexTTS2Options options) |
Creates a native, trainable IndexTTS2. |
IndexTTS2(NeuralNetworkArchitecture<T> architecture, string modelPath, IndexTTS2Options options) |
Creates an ONNX-backed IndexTTS2 for inference. |
Properties
| Property | Summary |
|---|---|
Tensor<T> EmotionReference { get; set; } |
A recording whose emotion to imitate (the release’s emo_audio_prompt), at EmotionReferenceSampleRate; null takes the speaker’s. |
bool TrainsOnSampleObjective { get; } |
|
double EmotionAlpha { get; set; } |
How strongly the emotion reference (or the weights) applies (emo_alpha, 1). |
double[] EmotionWeights { get; set; } |
Emotion weights, one per EmotionLabels (the release’s emo_vector); they mix the emotion table’s vectors and override the emotion reference. |
int EmotionReferenceSampleRate { get; set; } |
The sample rate of EmotionReference. |
int MaxTextLength { get; } |
|
string EmotionText { get; set; } |
A sentence whose emotion the text-to-emotion model reads into weights (use_emo_text). |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
T TrainTextToEmotion(string text, IReadOnlyList<double> distribution) |
One distillation step of the text-to-emotion model (§ Text-to-Emotion, Eq. |
Tensor<T> Synthesize(string text) |
|
TtsVoice<T> CreateVoice(Tensor<T> recording, int sampleRate) |
A voice from a recording of the speaker at sampleRate (the release’s prompt handling): the first 15 seconds, resampled by torchaudio to 16 kHz (its w2v-BERT features condition the GPT, their codes and CAM++ style condition the flow) and t… |
double[] DetectEmotion(string text) |
The text-to-emotion model’s emotion scores for text, one per EmotionLabels. |
int[] TokenizeText(string text) |
The token ids of text (normalized, CJK-split, BPE). |
void LoadEmotionTable(double[][] emotionRows, double[][] speakerRows) |
Sets the emotion table from tensors: the emotion vectors [rows, hidden] grouped by EmotionGroupSizes, and optionally the speaker vectors [rows, style]. |
void LoadReferenceWeights(IndexTts2Checkpoints checkpoints) |
Loads the release: the text-to-semantic model (gpt.pth; the text head, which inference never reads, is skipped), the semantic-to-mel module (s2mel.pth’s cfm, length_regulator and gpt_layer), w2v-BERT 2.0 with the feature statistic… |
void LoadTextToEmotion(string directory) |
Loads the text-to-emotion model’s model.safetensors from directory (a Qwen3ForCausalLM with tied embeddings). |