IndexTTS<T>
Models & Types · AiDotNet.TextToSpeech.CodecBased
IndexTTS (Deng et al. 2025): industrial zero-shot text-to-speech — a GPT-2 predicts a recording’s DVAE mel codes from BPE text tokens, conditioned on a conformer-perceiver encoding of a prompt recording, and BigVGAN turns the GPT’s latents into audio in the prompt speaker’s voice.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → IndexTTS<T>
For Beginners
Give IndexTTS a few seconds of someone speaking and it reads new text in that voice.
How It Works
The GPT reads [c, BT text ET, BM codes]: c, 32 latents a conformer and perceiver pool from the prompt’s
100-band log mel; the text, the release’s normalized and CJK-split BPE tokens; the codes, the DVAE’s (one per 1024
samples of 24 kHz audio). Generation is the release’s beam search; the latents of the generated codes, after long
silences are shrunk, go to BigVGAN with an ECAPA-TDNN embedding of the prompt’s mel added before and after every
upsampling.
The paper describes the system and its choices; its sizes, the codebook (a VQ DVAE in the release, where the
paper weighs FSQ) and the decoding come from the release (IndexTeam/Index-TTS), which the released weights need.
Neither publishes training code; training follows XTTS’s GPT recipe (IndexTTSOptions).
Example
var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new IndexTTS<float>(architecture, new IndexTTSOptions());
model.LoadReferenceWeights("Index-TTS"); // gpt.pth, bigvgan_generator.pth, dvae.pth, bpe.model
model.Voice = model.CreateVoice(promptAudio, promptSampleRate);
var audio = model.Synthesize("Text to speak in that voice."); // 24 kHz
Constructors
| Constructor | Summary |
|---|---|
IndexTTS(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
IndexTTS(NeuralNetworkArchitecture<T> architecture, IndexTTSOptions options) |
Creates a native, trainable IndexTTS. |
IndexTTS(NeuralNetworkArchitecture<T> architecture, string modelPath, IndexTTSOptions options) |
Creates an ONNX-backed IndexTTS for inference. |
Properties
| Property | Summary |
|---|---|
bool TrainsOnSampleObjective { get; } |
|
int MaxTextLength { get; } |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> Synthesize(string text) |
|
TtsVoice<T> CreateVoice(Tensor<T> recording, int sampleRate) |
A voice from a recording of the speaker at sampleRate (the release’s prompt handling: the channels averaged, resampled by torchaudio to 24 kHz, its log mel conditioning the GPT and BigVGAN). |
int[] EncodeToCodes(Tensor<T> audio) |
The DVAE’s codes of a recording at the model’s rate (get_codebook_indices of its log mel). |
int[] TokenizeText(string text) |
The token ids of text (normalized, CJK-split, BPE). |
void LoadReferenceWeights(string directory) |
Loads the release from its directory (IndexTeam/Index-TTS, or IndexTeam/IndexTTS-1.5 with Version1_5’s configuration): the GPT (gpt.pth), BigVGAN with its ECAPA-TDNN (bigvgan_generator.pth), the DVAE’s encoder and codebook (`dvae.… |