SPEARTTSOptions
Options & Configuration · AiDotNet.TextToSpeech.CodecBased
Options for SPEAR-TTS (Kharitonov et al. 2023, “Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision”).
Inheritance: ModelOptions → TtsModelOptions → SPEARTTSOptions
For Beginners
SPEAR-TTS learns to talk from hours of untranscribed audio and to read from only minutes of transcribed speech, then speaks in any voice it hears for three seconds.
How It Works
Tokens (§6.1, §7.1): semantic tokens are the 7th-layer w2v-BERT embeddings quantized to 512 clusters at 25 Hz, with consecutive repetitions removed; acoustic tokens are the first 3 RVQ levels (1024 codes each) of a 16 kHz SoundStream at 50 frames per second, interleaved per frame (150 tokens a second).
Reading, S1 (§4, §7.2): text (phonemes) to semantic tokens with T5-Large (24 encoder and 24 decoder layers,
width 1024, feed-forward 4096, 16 heads of 64). It is pretrained on semantic tokens to reconstruct each sequence
from a copy with every token deleted with probability 0.6, at dropout 0.5 (1M updates, batch 256), then finetuned.
Finetuning on parallel data updates the encoder’s embeddings and its lower ReadingTrainableEncoderLayers
with the rest frozen, at a dropout searched over {0.1, 0.3, 0.5} and label smoothing 0.1. Backtranslation
finetunes the same pretrained model the other way, semantic tokens to text, updating the top and bottom
BacktranslationTrainableDecoderLayers decoder layers. Every stage uses Adafactor with inverse
square-root decay. Synthesis reads with beam search (beam 10).
Speaking, S2 (§5, §7.3): semantic to acoustic tokens with a decoder-only Transformer (12 layers, 12 heads of 64, width 768, feed-forward 2048), trained on two non-overlapping windows of each utterance: the prompt’s semantic tokens, the target’s semantic tokens and the prompt’s acoustic tokens are the prefix, and the model learns the target’s acoustic tokens. Synthesis samples at temperature 0.75, prompted by 3 seconds of the speaker.
No official code exists. What the paper leaves open follows lucidrains/spear-tts-pytorch for S1’s procedure and
AudioLM’s sequence layout (AudioLmCore) for S2; see docs/text-to-speech/paper-fidelity-decisions.md.
Constructors
| Constructor | Summary |
|---|---|
SPEARTTSOptions() |
Creates the paper’s options. |
SPEARTTSOptions(SPEARTTSOptions other) |
Creates a copy of other. |
Properties
| Property | Summary |
|---|---|
SoundStreamOptions SoundStream { get; set; } |
The SoundStream codec. |
double LabelSmoothing { get; set; } |
Label smoothing of S1’s finetuning losses (0.1); pretraining uses none. |
double PretrainingDeletionProbability { get; set; } |
The probability of deleting each semantic token in the pretraining task (0.6). |
double PretrainingDropoutRate { get; set; } |
S1’s dropout during pretraining (0.5). |
double PromptSeconds { get; set; } |
The length of a voice prompt, and of the prompt window in S2’s training, in seconds (3). |
double SpeakingDropoutRate { get; set; } |
S2’s dropout. |
double Temperature { get; set; } |
S2’s sampling temperature (0.75, chosen on LJSpeech’s validation CER). |
int BacktranslationTrainableDecoderLayers { get; set; } |
N, the decoder’s top and bottom layers backtranslation finetuning updates (the paper searches {2, 3, 4, 12}; 2). |
int BeamSize { get; set; } |
S1’s beam size when reading (10). |
int CodebookSize { get; set; } |
Codes per RVQ level (1024). |
int HopSize { get; set; } |
The actual SoundStream stride product (320 samples in the paper). |
int MaxAcousticFrames { get; set; } |
The most codec frames S2 generates for one utterance. |
int MaxSemanticTokens { get; set; } |
The most semantic tokens S1 reads out for one text. |
int NumCodebooks { get; set; } |
SoundStream RVQ levels the acoustic tokens keep (3). |
int ReadingFeedForwardDim { get; set; } |
S1’s feed-forward width (T5-Large: 4096). |
int ReadingKeyValueDim { get; set; } |
S1’s per-head width (T5-Large: 64). |
int ReadingTrainableEncoderLayers { get; set; } |
The lower encoder layers finetuning on parallel data updates, with the encoder’s embeddings (the paper searches {4, 6, 8} on ground-truth data and finetunes the lower 4 last; 4). |
int SampleRate { get; set; } |
The actual SoundStream audio sample rate (16 kHz in the paper). |
int SamplingSeed { get; set; } |
The seed of synthesis and of the training corruption. |
int SemanticCodes { get; set; } |
Semantic token vocabulary: k-means clusters of w2v-BERT’s 7th layer (512). |
int SemanticFrameRate { get; set; } |
Semantic tokens per second before repetitions are removed (25). |
int SpeakingFeedForwardDim { get; set; } |
S2’s feed-forward width (2048). |
int SpeakingHeads { get; set; } |
S2’s attention heads (12). |
int SpeakingKeyValueDim { get; set; } |
S2’s per-head width (64). |
int SpeakingLayers { get; set; } |
S2’s layers (12). |
int SpeakingModelDim { get; set; } |
S2’s width (768). |