ParlerTTS<T>
Models & Types · AiDotNet.TextToSpeech.DescriptionBased
Parler-TTS: speech in the voice a plain-language description gives (Lyth & King 2024), as the open reproduction huggingface/parler-tts builds it.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → ParlerTTS<T>
Implements: ICodecTts<T>
For Beginners
Instead of a recording of a speaker, you describe the voice you want in plain English (“A female speaker delivers a slightly expressive and animated speech with a moderate speed and pitch”) and the model speaks the text in such a voice.
How It Works
The paper’s model (§3): a decoder-only Transformer predicts the codes of DAC (44.1 kHz, nine codebooks) in MusicGen’s
delay pattern, codebook q delayed by q steps. The transcript’s token embeddings are prepended to the
codes, and the description is read by cross-attention from a frozen Flan-T5 encoder. The rest follows the
reproduction (ParlerTTSForConditionalGeneration): per-codebook code embeddings summed, sinusoidal positions,
pre-norm layers with bias-free projections and the exact GELU, a bias-free head per codebook, the cross-entropy of
each codebook averaged over its positions and the codebooks, and generation that lets codebook q end only after
codebook q − 1 has.
Both texts are tokenized with T5’s SentencePiece vocabulary, the transcript into the model’s input
(Synthesize) and the description into the voice (CreateVoice).
Training reads the transcript (Tokens), the description
(Description) and the recording’s DAC codes
(CodecTokens, or the recording, which the model encodes).
Constructors
| Constructor | Summary |
|---|---|
ParlerTTS(NeuralNetworkArchitecture<T> architecture, ParlerTTSOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a native, trainable Parler-TTS. |
ParlerTTS(NeuralNetworkArchitecture<T> architecture, string modelPath, ParlerTTSOptions options) |
Creates a native, trainable Parler-TTS. |
Properties
| Property | Summary |
|---|---|
DAC<T> Codec { get; } |
The DAC whose codes the model predicts (load its released weights with LoadPretrainedWeights). |
int CodebookSize { get; } |
|
int CodecFrameRate { get; } |
|
int CodecTokenCodebooks { get; } |
|
int CodecTokenVocabulary { get; } |
|
int MaxTextLength { get; } |
|
int NumCodebooks { get; } |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> DecodeFromTokens(Tensor<T> tokens) |
|
Tensor<T> EncodeToTokens(Tensor<T> audio) |
|
TtsVoice<T> CreateVoice(string description) |
A voice for Voice: a plain-language description of the speaker, the recording conditions and the delivery. |
int[] Tokenize(string text) |
The T5 token ids of text, ending with </s>, as the reference’s tokenizer gives them. |
void LoadReferenceWeights(string path) |
Loads a ParlerTTSForConditionalGeneration state dict (safetensors or .pt): the decoder, the transcript embedding, the projection and the Flan-T5 encoder (text_encoder.*). |