TortoiseTTS<T>
Models & Types · AiDotNet.TextToSpeech.CodecBased
TorToiSe: Better Speech Synthesis Through Scaling.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → TortoiseTTS<T>
Implements: ICodecTts<T>
How It Works
The native graph is the complete released pipeline: an 8,192-code VQ-VAE speech tokenizer, a separately positioned GPT-2 speech prior, CLVP candidate ranking, latent-conditioned classifier-free diffusion, and UnivNet-c32. The components are deliberately not collapsed into one generic transformer because they are trained independently and perform different jobs in the paper.
Constructors
| Constructor | Summary |
|---|---|
TortoiseTTS(NeuralNetworkArchitecture<T> architecture, TortoiseTTSOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates the native paper/release graph. |
TortoiseTTS(NeuralNetworkArchitecture<T> architecture, string modelPath, TortoiseTTSOptions options) |
Creates the native paper/release graph. |
Properties
| Property | Summary |
|---|---|
bool HasPaperPipeline { get; } |
Whether this instance owns all five native TorToiSe inference components. |
bool TrainsOnSampleObjective { get; } |
|
int CodebookSize { get; } |
|
int CodecFrameRate { get; } |
|
int CodecTokenCodebooks { get; } |
|
int CodecTokenVocabulary { get; } |
|
int MaxTextLength { get; } |
|
int NumCodebooks { get; } |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
T TrainClvpBatch(IReadOnlyList<TtsTrainingSample<T>> samples) |
Trains CLVP with the released symmetric in-batch contrastive objective. |
Tensor<T> DecodeFromTokens(Tensor<T> tokens) |
Decodes VQ-VAE codes through its mel decoder and the released UnivNet vocoder. |
Tensor<T> EncodeToTokens(Tensor<T> audio) |
Encodes 24 kHz audio to the paper’s one-stream VQ-VAE speech codes. |
Tensor<T> EncodeToTokens(Tensor<T> audio, int sampleRate) |
Encodes audio at an explicitly supplied rate to VQ-VAE speech codes. |
Tensor<T> Synthesize(string text) |
Runs autoregressive candidate generation, CLVP ranking, DDIM and UnivNet. |
Tensor<T> SynthesizeWithVoice(string text, Tensor<T> recording, int sampleRate) |
Synthesizes with a recording without changing Voice. |
Tensor<T> TokenizeText(string text) |
Runs the released English cleaner and exact BPE vocabulary. |
TtsVoice<T> CreateVoice(Tensor<T> recording, int sampleRate) |
Creates a zero-shot voice from a recording and resamples it to TorToiSe’s 24 kHz output rate. |
double ScoreCandidate(Tensor<T> textTokens, Tensor<T> speechTokens) |
Returns CLVP’s scalar compatibility score for a candidate speech-code sequence. |
void LoadAuthorWeights(string autoregressivePath, string clvpPath, string diffusionPath, string speechTokenizerPath, string vocoderPath, string textTokenizerPath, string melNormalizationPath, string vocoderPrefix) |
Loads all five original TorToiSe component checkpoints without a parameter-vector converter. |
void LoadAutoregressiveWeights(string path) |
Loads the authors’ original named autoregressive.pth state dictionary. |
void LoadClvpWeights(string path) |
Loads the authors’ original named clvp2.pth state dictionary. |
void LoadConvertedWeights(string path) |
Copies {ConvertedParameterTensor} from source straight into a parameter vector. |
void LoadDiffusionWeights(string path) |
Loads the authors’ original named diffusion_decoder.pth state dictionary. |
void LoadMelNormalization(string path) |
Loads a custom original mel_norms.pth tensor for the autoregressive front end. |
void LoadTextTokenizer(string path) |
Loads the original Whitespace/character-BPE tokenizer for a checkpoint. |
void LoadTokenizerWeights(string path) |
Loads the authors’ original dVAE tensors, including EMA counts and codebook sums. |
void LoadVocoderWeights(string path, string prefix) |
Loads vocoder.pth’s original model_g state, or an extracted UnivNet state with an empty prefix. |