IndexTTS2<T>

Models & Types · AiDotNet.TextToSpeech.Latest

IndexTTS2 (Zhou et al. 2025): zero-shot autoregressive text-to-speech with precise duration control and emotion control separated from the speaker’s timbre — text to semantic tokens by a GPT-2 conditioned on a speaker prompt and a style prompt, semantic tokens (with the GPT’s latents) to a mel spectrogram by flow matching, and BigVGAN-v2 to audio.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → IndexTTS2<T>

For Beginners

Give IndexTTS2 a few seconds of someone speaking and it reads new text in that voice. You can also give it a second recording whose emotion to copy, a set of emotion weights, or a sentence describing the emotion, and say how many tokens (roughly how long) the speech should be.

How It Works

The text-to-semantic model reads [c + e, p, BT text ET, BA semantic]: c, 32 latents a conformer and perceiver pool from the speaker prompt’s w2v-BERT 2.0 features; e, the emotion vector pooled the same way from a style prompt (or mixed from an emotion table by weights, or by weights a text-to-emotion model reads off a sentence); p, the duration token — by default the paper’s W_sem h(T) naming the number of semantic tokens to generate, or zeros to generate freely. The semantic-to-mel module conditions seed-vc’s DiT on the semantic codec’s features plus the GPT’s last hidden states (the paper’s GPT latent enhancement), the prompt’s mel and its CAM++ style.

The paper leaves most sizes to the release (IndexTeam/IndexTTS-2), whose code fills the gaps; where the two differ the paper is the default and the release an option (Release), which the released weights need.

Example

var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var options = IndexTTS2Options.Release();
options.BpeModelPath = "IndexTTS-2/bpe.model";
var model = new IndexTTS2<float>(architecture, options);
model.LoadReferenceWeights(new IndexTts2Checkpoints { Directory = "IndexTTS-2", SemanticModel = "w2v-bert-2.0/model.safetensors",
    SemanticCodec = "MaskGCT/semantic_codec/model.safetensors", SpeakerEncoder = "campplus_cn_common.bin",
    Vocoder = "bigvgan_v2_22khz_80band_256x/bigvgan_generator.pt" });
model.Voice = model.CreateVoice(promptAudio, promptSampleRate);
var audio = model.Synthesize("Text to speak in that voice.");   // 22.05 kHz

Constructors

Constructor Summary
IndexTTS2(NeuralNetworkArchitecture<T> architecture) Calls the fully specified overload with the omitted parameters’ defaults.
IndexTTS2(NeuralNetworkArchitecture<T> architecture, IndexTTS2Options options) Creates a native, trainable IndexTTS2.
IndexTTS2(NeuralNetworkArchitecture<T> architecture, string modelPath, IndexTTS2Options options) Creates an ONNX-backed IndexTTS2 for inference.

Properties

Property Summary
Tensor<T> EmotionReference { get; set; } A recording whose emotion to imitate (the release’s emo_audio_prompt), at EmotionReferenceSampleRate; null takes the speaker’s.
bool TrainsOnSampleObjective { get; }
double EmotionAlpha { get; set; } How strongly the emotion reference (or the weights) applies (emo_alpha, 1).
double[] EmotionWeights { get; set; } Emotion weights, one per EmotionLabels (the release’s emo_vector); they mix the emotion table’s vectors and override the emotion reference.
int EmotionReferenceSampleRate { get; set; } The sample rate of EmotionReference.
int MaxTextLength { get; }
string EmotionText { get; set; } A sentence whose emotion the text-to-emotion model reads into weights (use_emo_text).

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
T TrainTextToEmotion(string text, IReadOnlyList<double> distribution) One distillation step of the text-to-emotion model (§ Text-to-Emotion, Eq.
Tensor<T> Synthesize(string text)
TtsVoice<T> CreateVoice(Tensor<T> recording, int sampleRate) A voice from a recording of the speaker at sampleRate (the release’s prompt handling): the first 15 seconds, resampled by torchaudio to 16 kHz (its w2v-BERT features condition the GPT, their codes and CAM++ style condition the flow) and t…
double[] DetectEmotion(string text) The text-to-emotion model’s emotion scores for text, one per EmotionLabels.
int[] TokenizeText(string text) The token ids of text (normalized, CJK-split, BPE).
void LoadEmotionTable(double[][] emotionRows, double[][] speakerRows) Sets the emotion table from tensors: the emotion vectors [rows, hidden] grouped by EmotionGroupSizes, and optionally the speaker vectors [rows, style].
void LoadReferenceWeights(IndexTts2Checkpoints checkpoints) Loads the release: the text-to-semantic model (gpt.pth; the text head, which inference never reads, is skipped), the semantic-to-mel module (s2mel.pth’s cfm, length_regulator and gpt_layer), w2v-BERT 2.0 with the feature statistic…
void LoadTextToEmotion(string directory) Loads the text-to-emotion model’s model.safetensors from directory (a Qwen3ForCausalLM with tied embeddings).