ParlerTTSOptions

Options & Configuration · AiDotNet.TextToSpeech.DescriptionBased

Options for Parler-TTS (Lyth & King 2024, “Natural language guidance of high-fidelity text-to-speech with synthetic annotations”).

Inheritance: ModelOptions → TtsModelOptions → ParlerTTSOptions

For Beginners

These options set the sizes of Parler-TTS’s networks, its codec and how it is trained and sampled. The defaults reproduce the released model.

How It Works

The paper fixes the design (§3): the transcript prepended to the codes of DAC at 44.1 kHz (nine codebooks, 86 frames a second) in MusicGen’s delay pattern, and the description read by cross-attention from a frozen Flan-T5 encoder. It leaves sizes and training settings open; the defaults take them from the open reproduction the paper’s code release points to (huggingface/parler-tts), as its released parler-tts-mini-v1 configures them: a 24-layer, 1024-wide decoder with 16 heads and feed-forward 4096, sinusoidal positions, dropout 0.1, Flan-T5-large as the description encoder, and training with AdamW (9.5e-4, betas 0.9 and 0.99, weight decay 0.01, gradients clipped to norm 1) held constant after 20,000 warm-up updates, the encoder frozen. Generation samples (temperature 1, top 50) for at most 2580 steps and at least 10.

Constructors

Constructor Summary
ParlerTTSOptions() Creates the released parler-tts-mini-v1 configuration.
ParlerTTSOptions(ParlerTTSOptions other) Creates a copy of other.

Properties

Property Summary
DACOptions Codec { get; set; } The codec (DAC 44.1 kHz at 8 kbps: nine codebooks).
bool DoSample { get; set; } Whether generation samples (on) or takes the most likely code.
bool FreezeTextEncoder { get; set; } Whether the description encoder is frozen in training (on, §3).
bool PromptCrossAttention { get; set; } Whether the transcript is cross-attended next to the description instead of prepended to the codes (off, as the paper prepends it; the reproduction’s prompt_cross_attention).
bool RotaryEmbeddings { get; set; } Rotary position embeddings in place of the added sinusoidal ones (off in the released model; the reproduction’s rope_embeddings).
double ActivationDropout { get; set; } Dropout after the feed-forward activation (0).
double AttentionDropout { get; set; } Dropout on the attention weights (0).
double Beta1 { get; set; } AdamW β₁ (0.9).
double Beta2 { get; set; } AdamW β₂ (0.99).
double Epsilon { get; set; } AdamW ε (1e-8).
double InitializerFactor { get; set; } Standard deviation of the decoder’s initial weights (0.02, initializer_factor).
double LayerDrop { get; set; } Probability of skipping a decoder layer in training (0).
double MaxGradientNorm { get; set; } Gradient-norm clipping (1; 0 turns it off).
double RopeTheta { get; set; } The rotary embeddings’ base (10,000).
double Temperature { get; set; } Sampling temperature (1).
double TextEncoderDropout { get; set; } Dropout inside the description encoder (0.1, Flan-T5’s; active in training even when frozen, as the reproduction leaves the frozen encoder in training mode).
double[] CodebookWeights { get; set; } Per-codebook loss weights (none: the mean over the codebooks).
int AudioVocabulary { get; set; } Logits per codebook (1088): the codes, the end token (the codebook size) and the start token (one more), rounded up as released.
int CodebookSize { get; set; } Codes per codebook (1024, the codec’s).
int CrossAttentionKeyValueHeads { get; set; } Key/value heads of the decoder’s cross-attention (0: as many as KeyValueHeads).
int FeedForwardDim { get; set; } Feed-forward width of the decoder (4096).
int KeyValueHeads { get; set; } Key/value heads of the decoder’s self-attention (0: one per head, as released; fewer is grouped-query attention).
int MaxLength { get; set; } Most decoder steps, the start column included (2580: 30 seconds of codes and the delay).
int MinNewTokens { get; set; } Steps before the end token is allowed (10).
int NumCodebooks { get; set; } Codebooks (9: DAC at 8 kbps).
int PromptVocabulary { get; set; } Rows of the transcript embedding (32,128: Flan-T5’s vocabulary, which tokenizes the transcript).
int SamplingSeed { get; set; } Seed of the training draws (dropout) and of sampling.
int TextEncoderDim { get; set; } Width of the description encoder (1024); a width other than the decoder’s adds a projection.
int TextEncoderFeedForwardDim { get; set; } Feed-forward width of the description encoder (2816, gated GELU).
int TextEncoderHeads { get; set; } Attention heads of the description encoder (16).
int TextEncoderKeyValueDim { get; set; } Width of each attention head of the description encoder (64).
int TextEncoderLayers { get; set; } Layers of the description encoder (24).
int TextEncoderVocabulary { get; set; } Vocabulary of the description encoder (32,128).
int TopK { get; set; } Sampling top-k cut (50; 0 keeps every code).
int WarmupSteps { get; set; } Linear warm-up of the learning rate, in updates (20,000), after which it stays constant.