XTTSv2<T>

Models & Types · AiDotNet.TextToSpeech.VoiceCloning

XTTS-v2: multilingual zero-shot TTS with a dVAE tokenizer, GPT-2 and speaker-conditioned HiFi-GAN.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → XTTSv2<T>

Implements: ICodecTts<T>

Known derived types: XTTSv2Clone<T>

How It Works

This is the architecture released by Coqui for Casanova et al. (2024): 80-band 22.05 kHz log-power mels feed a two-stride, 1024-code dVAE tokenizer; six Tortoise attention blocks and a 32-latent Perceiver encode a reference; a 30-layer GPT-2 predicts acoustic ids; its teacher-forced hidden states—not reconstructed dVAE codes—condition a 24 kHz HiFi-GAN. Speaker identity is the frozen 512-wide H/ASP SE-ResNet embedding added before and after every HiFi-GAN upsampler. The public training path implements the released GPT fine-tuning objective (text CE × .01 plus acoustic CE) while the dVAE, speaker encoder and waveform decoder act as pretrained feature modules.

Constructors

Constructor Summary
XTTSv2(NeuralNetworkArchitecture<T> architecture, XTTSv2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates the trainable native XTTS-v2 graph.
XTTSv2(NeuralNetworkArchitecture<T> architecture, string modelPath, XTTSv2Options options) Creates the trainable native XTTS-v2 graph.

Properties

Property Summary
bool HasPaperPipeline { get; } Whether the native dVAE, conditioning/GPT, H/ASP and HiFi-GAN components are present.
bool TokenizerLoaded { get; }
bool TrainsOnSampleObjective { get; }
int CodebookSize { get; }
int CodecFrameRate { get; }
int CodecTokenCodebooks { get; }
int CodecTokenVocabulary { get; }
int ConditioningAttentionBlocks { get; }
int ConditioningLatentCount { get; }
int DvaeDownsampleFactor { get; }
int MaxTextLength { get; }
int NumCodebooks { get; }
int TransformerLayerCount { get; }

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> DecodeFromTokens(Tensor<T> tokens) Reconstructs diagnostic audio from the dVAE mel decoder with mel-filter inversion and Griffin-Lim.
Tensor<T> DecodeTokensToMel(Tensor<T> tokens) Decodes dVAE ids to their normalized log-power mel reconstruction.
Tensor<T> EncodeToTokens(Tensor<T> audio) Encodes a 24 kHz waveform with the released 22.05 kHz dVAE tokenizer.
Tensor<T> ExtractConditioningLatents(Tensor<T> referenceAudio) Computes the averaged 32×1024 conditioning sequence from six-second chunks.
Tensor<T> ExtractSpeakerEmbedding(Tensor<T> referenceAudio) Returns the released H/ASP speaker embedding, L2-normalized to 512 dimensions.
Tensor<T> Synthesize(string text) Synthesizes in Voice’s reference voice.
Tensor<T> Synthesize(string text, string language, Tensor<T> referenceAudio) Runs the released zero-shot path for a language and 24 kHz reference recording.
Tensor<T> SynthesizeTokens(Tensor<T> textTokens, Tensor<T> referenceAudio) Runs the native path from already-tokenized XTTS BPE ids (useful for dataset and converted-tokenizer integrations).
TtsVoice<T> CreateVoice(Tensor<T> referenceAudio) Creates the voice value consumed by Synthesize.
int[] TokenizeText(string text, string language) Returns the released language-prefixed BPE ids.
string NormalizeText(string text, string language) Applies the release’s lowercasing/space cleanup and East-Asian romanization boundary.
void LoadAuthorWeights(string modelCheckpoint, string dvaeCheckpoint, string tokenizerPath, string melStatsPath) Loads named Coqui model/dVAE tensors and their original tokenizer and mel statistics.
void LoadConvertedWeights(string path) Copies {ConvertedParameterTensor} from source straight into a parameter vector.
void LoadMelStats(string path) Loads the 80 released mel normalization values from mel_stats.pth or safetensors.
void LoadReferenceWeights(string convertedCheckpointPath, string tokenizerPath, string melStatsPath) Loads a converted official graph plus its unmodified tokenizer and mel statistics.
void LoadTokenizer(string path) Loads the exact BPE tokenizer shipped by coqui/XTTS-v2.