TortoiseTTS<T>

Models & Types · AiDotNet.TextToSpeech.CodecBased

TorToiSe: Better Speech Synthesis Through Scaling.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → TortoiseTTS<T>

Implements: ICodecTts<T>

How It Works

The native graph is the complete released pipeline: an 8,192-code VQ-VAE speech tokenizer, a separately positioned GPT-2 speech prior, CLVP candidate ranking, latent-conditioned classifier-free diffusion, and UnivNet-c32. The components are deliberately not collapsed into one generic transformer because they are trained independently and perform different jobs in the paper.

Constructors

Constructor Summary
TortoiseTTS(NeuralNetworkArchitecture<T> architecture, TortoiseTTSOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates the native paper/release graph.
TortoiseTTS(NeuralNetworkArchitecture<T> architecture, string modelPath, TortoiseTTSOptions options) Creates the native paper/release graph.

Properties

Property Summary
bool HasPaperPipeline { get; } Whether this instance owns all five native TorToiSe inference components.
bool TrainsOnSampleObjective { get; }
int CodebookSize { get; }
int CodecFrameRate { get; }
int CodecTokenCodebooks { get; }
int CodecTokenVocabulary { get; }
int MaxTextLength { get; }
int NumCodebooks { get; }

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
T TrainClvpBatch(IReadOnlyList<TtsTrainingSample<T>> samples) Trains CLVP with the released symmetric in-batch contrastive objective.
Tensor<T> DecodeFromTokens(Tensor<T> tokens) Decodes VQ-VAE codes through its mel decoder and the released UnivNet vocoder.
Tensor<T> EncodeToTokens(Tensor<T> audio) Encodes 24 kHz audio to the paper’s one-stream VQ-VAE speech codes.
Tensor<T> EncodeToTokens(Tensor<T> audio, int sampleRate) Encodes audio at an explicitly supplied rate to VQ-VAE speech codes.
Tensor<T> Synthesize(string text) Runs autoregressive candidate generation, CLVP ranking, DDIM and UnivNet.
Tensor<T> SynthesizeWithVoice(string text, Tensor<T> recording, int sampleRate) Synthesizes with a recording without changing Voice.
Tensor<T> TokenizeText(string text) Runs the released English cleaner and exact BPE vocabulary.
TtsVoice<T> CreateVoice(Tensor<T> recording, int sampleRate) Creates a zero-shot voice from a recording and resamples it to TorToiSe’s 24 kHz output rate.
double ScoreCandidate(Tensor<T> textTokens, Tensor<T> speechTokens) Returns CLVP’s scalar compatibility score for a candidate speech-code sequence.
void LoadAuthorWeights(string autoregressivePath, string clvpPath, string diffusionPath, string speechTokenizerPath, string vocoderPath, string textTokenizerPath, string melNormalizationPath, string vocoderPrefix) Loads all five original TorToiSe component checkpoints without a parameter-vector converter.
void LoadAutoregressiveWeights(string path) Loads the authors’ original named autoregressive.pth state dictionary.
void LoadClvpWeights(string path) Loads the authors’ original named clvp2.pth state dictionary.
void LoadConvertedWeights(string path) Copies {ConvertedParameterTensor} from source straight into a parameter vector.
void LoadDiffusionWeights(string path) Loads the authors’ original named diffusion_decoder.pth state dictionary.
void LoadMelNormalization(string path) Loads a custom original mel_norms.pth tensor for the autoregressive front end.
void LoadTextTokenizer(string path) Loads the original Whitespace/character-BPE tokenizer for a checkpoint.
void LoadTokenizerWeights(string path) Loads the authors’ original dVAE tensors, including EMA counts and codebook sums.
void LoadVocoderWeights(string path, string prefix) Loads vocoder.pth’s original model_g state, or an extracted UnivNet state with an empty prefix.