ParlerTTS<T>

Models & Types · AiDotNet.TextToSpeech.DescriptionBased

Parler-TTS: speech in the voice a plain-language description gives (Lyth & King 2024), as the open reproduction huggingface/parler-tts builds it.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → ParlerTTS<T>

Implements: ICodecTts<T>

For Beginners

Instead of a recording of a speaker, you describe the voice you want in plain English (“A female speaker delivers a slightly expressive and animated speech with a moderate speed and pitch”) and the model speaks the text in such a voice.

How It Works

The paper’s model (§3): a decoder-only Transformer predicts the codes of DAC (44.1 kHz, nine codebooks) in MusicGen’s delay pattern, codebook q delayed by q steps. The transcript’s token embeddings are prepended to the codes, and the description is read by cross-attention from a frozen Flan-T5 encoder. The rest follows the reproduction (ParlerTTSForConditionalGeneration): per-codebook code embeddings summed, sinusoidal positions, pre-norm layers with bias-free projections and the exact GELU, a bias-free head per codebook, the cross-entropy of each codebook averaged over its positions and the codebooks, and generation that lets codebook q end only after codebook q − 1 has.

Both texts are tokenized with T5’s SentencePiece vocabulary, the transcript into the model’s input (Synthesize) and the description into the voice (CreateVoice). Training reads the transcript (Tokens), the description (Description) and the recording’s DAC codes (CodecTokens, or the recording, which the model encodes).

Constructors

Constructor Summary
ParlerTTS(NeuralNetworkArchitecture<T> architecture, ParlerTTSOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a native, trainable Parler-TTS.
ParlerTTS(NeuralNetworkArchitecture<T> architecture, string modelPath, ParlerTTSOptions options) Creates a native, trainable Parler-TTS.

Properties

Property Summary
DAC<T> Codec { get; } The DAC whose codes the model predicts (load its released weights with LoadPretrainedWeights).
int CodebookSize { get; }
int CodecFrameRate { get; }
int CodecTokenCodebooks { get; }
int CodecTokenVocabulary { get; }
int MaxTextLength { get; }
int NumCodebooks { get; }

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> DecodeFromTokens(Tensor<T> tokens)
Tensor<T> EncodeToTokens(Tensor<T> audio)
TtsVoice<T> CreateVoice(string description) A voice for Voice: a plain-language description of the speaker, the recording conditions and the delivery.
int[] Tokenize(string text) The T5 token ids of text, ending with </s>, as the reference’s tokenizer gives them.
void LoadReferenceWeights(string path) Loads a ParlerTTSForConditionalGeneration state dict (safetensors or .pt): the decoder, the transcript embedding, the projection and the Flan-T5 encoder (text_encoder.*).