XTTSv2<T>
Models & Types · AiDotNet.TextToSpeech.VoiceCloning
XTTS-v2: multilingual zero-shot TTS with a dVAE tokenizer, GPT-2 and speaker-conditioned HiFi-GAN.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → XTTSv2<T>
Implements: ICodecTts<T>
Known derived types: XTTSv2Clone<T>
How It Works
This is the architecture released by Coqui for Casanova et al. (2024): 80-band 22.05 kHz log-power mels feed a two-stride, 1024-code dVAE tokenizer; six Tortoise attention blocks and a 32-latent Perceiver encode a reference; a 30-layer GPT-2 predicts acoustic ids; its teacher-forced hidden states—not reconstructed dVAE codes—condition a 24 kHz HiFi-GAN. Speaker identity is the frozen 512-wide H/ASP SE-ResNet embedding added before and after every HiFi-GAN upsampler. The public training path implements the released GPT fine-tuning objective (text CE × .01 plus acoustic CE) while the dVAE, speaker encoder and waveform decoder act as pretrained feature modules.
Constructors
| Constructor | Summary |
|---|---|
XTTSv2(NeuralNetworkArchitecture<T> architecture, XTTSv2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates the trainable native XTTS-v2 graph. |
XTTSv2(NeuralNetworkArchitecture<T> architecture, string modelPath, XTTSv2Options options) |
Creates the trainable native XTTS-v2 graph. |
Properties
| Property | Summary |
|---|---|
bool HasPaperPipeline { get; } |
Whether the native dVAE, conditioning/GPT, H/ASP and HiFi-GAN components are present. |
bool TokenizerLoaded { get; } |
|
bool TrainsOnSampleObjective { get; } |
|
int CodebookSize { get; } |
|
int CodecFrameRate { get; } |
|
int CodecTokenCodebooks { get; } |
|
int CodecTokenVocabulary { get; } |
|
int ConditioningAttentionBlocks { get; } |
|
int ConditioningLatentCount { get; } |
|
int DvaeDownsampleFactor { get; } |
|
int MaxTextLength { get; } |
|
int NumCodebooks { get; } |
|
int TransformerLayerCount { get; } |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> DecodeFromTokens(Tensor<T> tokens) |
Reconstructs diagnostic audio from the dVAE mel decoder with mel-filter inversion and Griffin-Lim. |
Tensor<T> DecodeTokensToMel(Tensor<T> tokens) |
Decodes dVAE ids to their normalized log-power mel reconstruction. |
Tensor<T> EncodeToTokens(Tensor<T> audio) |
Encodes a 24 kHz waveform with the released 22.05 kHz dVAE tokenizer. |
Tensor<T> ExtractConditioningLatents(Tensor<T> referenceAudio) |
Computes the averaged 32×1024 conditioning sequence from six-second chunks. |
Tensor<T> ExtractSpeakerEmbedding(Tensor<T> referenceAudio) |
Returns the released H/ASP speaker embedding, L2-normalized to 512 dimensions. |
Tensor<T> Synthesize(string text) |
Synthesizes in Voice’s reference voice. |
Tensor<T> Synthesize(string text, string language, Tensor<T> referenceAudio) |
Runs the released zero-shot path for a language and 24 kHz reference recording. |
Tensor<T> SynthesizeTokens(Tensor<T> textTokens, Tensor<T> referenceAudio) |
Runs the native path from already-tokenized XTTS BPE ids (useful for dataset and converted-tokenizer integrations). |
TtsVoice<T> CreateVoice(Tensor<T> referenceAudio) |
Creates the voice value consumed by Synthesize. |
int[] TokenizeText(string text, string language) |
Returns the released language-prefixed BPE ids. |
string NormalizeText(string text, string language) |
Applies the release’s lowercasing/space cleanup and East-Asian romanization boundary. |
void LoadAuthorWeights(string modelCheckpoint, string dvaeCheckpoint, string tokenizerPath, string melStatsPath) |
Loads named Coqui model/dVAE tensors and their original tokenizer and mel statistics. |
void LoadConvertedWeights(string path) |
Copies {ConvertedParameterTensor} from source straight into a parameter vector. |
void LoadMelStats(string path) |
Loads the 80 released mel normalization values from mel_stats.pth or safetensors. |
void LoadReferenceWeights(string convertedCheckpointPath, string tokenizerPath, string melStatsPath) |
Loads a converted official graph plus its unmodified tokenizer and mel statistics. |
void LoadTokenizer(string path) |
Loads the exact BPE tokenizer shipped by coqui/XTTS-v2. |