IndexTTS<T>

Models & Types · AiDotNet.TextToSpeech.CodecBased

IndexTTS (Deng et al. 2025): industrial zero-shot text-to-speech — a GPT-2 predicts a recording’s DVAE mel codes from BPE text tokens, conditioned on a conformer-perceiver encoding of a prompt recording, and BigVGAN turns the GPT’s latents into audio in the prompt speaker’s voice.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → IndexTTS<T>

For Beginners

Give IndexTTS a few seconds of someone speaking and it reads new text in that voice.

How It Works

The GPT reads [c, BT text ET, BM codes]: c, 32 latents a conformer and perceiver pool from the prompt’s 100-band log mel; the text, the release’s normalized and CJK-split BPE tokens; the codes, the DVAE’s (one per 1024 samples of 24 kHz audio). Generation is the release’s beam search; the latents of the generated codes, after long silences are shrunk, go to BigVGAN with an ECAPA-TDNN embedding of the prompt’s mel added before and after every upsampling.

The paper describes the system and its choices; its sizes, the codebook (a VQ DVAE in the release, where the paper weighs FSQ) and the decoding come from the release (IndexTeam/Index-TTS), which the released weights need. Neither publishes training code; training follows XTTS’s GPT recipe (IndexTTSOptions).

Example

var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new IndexTTS<float>(architecture, new IndexTTSOptions());
model.LoadReferenceWeights("Index-TTS");             // gpt.pth, bigvgan_generator.pth, dvae.pth, bpe.model
model.Voice = model.CreateVoice(promptAudio, promptSampleRate);
var audio = model.Synthesize("Text to speak in that voice.");   // 24 kHz

Constructors

Constructor Summary
IndexTTS(NeuralNetworkArchitecture<T> architecture) Calls the fully specified overload with the omitted parameters’ defaults.
IndexTTS(NeuralNetworkArchitecture<T> architecture, IndexTTSOptions options) Creates a native, trainable IndexTTS.
IndexTTS(NeuralNetworkArchitecture<T> architecture, string modelPath, IndexTTSOptions options) Creates an ONNX-backed IndexTTS for inference.

Properties

Property Summary
bool TrainsOnSampleObjective { get; }
int MaxTextLength { get; }

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> Synthesize(string text)
TtsVoice<T> CreateVoice(Tensor<T> recording, int sampleRate) A voice from a recording of the speaker at sampleRate (the release’s prompt handling: the channels averaged, resampled by torchaudio to 24 kHz, its log mel conditioning the GPT and BigVGAN).
int[] EncodeToCodes(Tensor<T> audio) The DVAE’s codes of a recording at the model’s rate (get_codebook_indices of its log mel).
int[] TokenizeText(string text) The token ids of text (normalized, CJK-split, BPE).
void LoadReferenceWeights(string directory) Loads the release from its directory (IndexTeam/Index-TTS, or IndexTeam/IndexTTS-1.5 with Version1_5’s configuration): the GPT (gpt.pth), BigVGAN with its ECAPA-TDNN (bigvgan_generator.pth), the DVAE’s encoder and codebook (`dvae.…