PhemeOptions
Options & Configuration · AiDotNet.TextToSpeech.CodecBased
Options for Pheme (Budzianowski et al. 2024, “Pheme: Efficient and Conversational Speech Generation”).
Inheritance: ModelOptions → TtsModelOptions → PhemeOptions
For Beginners
These options set the sizes and training settings of Pheme’s two networks. The defaults reproduce the released small model.
How It Works
Pheme is a two-stage model over SpeechTokenizer codes: a T5 encoder–decoder turns phonemes into semantic tokens (the first SpeechTokenizer codebook), and a SoundStorm-style Conformer fills in the seven acoustic codebooks non-autoregressively, conditioned on the semantic tokens, a prompt and a speaker embedding.
The defaults are the paper’s small model (Table 5: a 6 + 6-layer T5 of width 512, a 3-layer acoustic Conformer of
width 1024) and its training recipe (§4.1: both networks trained with AdamW, β = 0.9 / 0.98, at 5e-4 with 10,000
warm-up steps and linear decay to zero over 800,000 steps; the paper also writes “decayed from 2×10⁻⁴”, which
contradicts its stated rate, so the stated 5e-4 is the peak). Large is the paper’s large model. The
released checkpoints differ from the paper’s text: OfficialSmallCheckpoint (PolyAI/pheme_small) has a
768-wide acoustic model, and OfficialLargeCheckpoint (PolyAI/pheme) no speaker embedding.
What the paper leaves to its code follows the reference implementation (PolyAI-LDN/pheme): the text-to-semantic
model is Hugging Face’s T5ForConditionalGeneration v1.0 (d_ff 2048, d_kv 64, 8 heads, ReLU, dropout 0.1,
32 relative buckets up to distance 128), with the Trainer’s defaults where the paper is silent (no weight decay,
gradient norm clipped to 1); the acoustic model is lucidrains’ SoundStorm Conformer (8 heads of 64, feed-forward ×4,
convolution expansion 2, kernel 5, dropout 0.1) with a 512 → width projection of the L2-normalized pyannote speaker
embedding (dropout 0.05) and PyTorch’s default AdamW weight decay 0.01; codes are SpeechTokenizer’s (1024 semantic
and 1024 acoustic codes, 7 acoustic codebooks, 16 kHz, 50 frames per second). Synthesis samples semantic tokens at
temperature 0.7 from the top 210 (§4.1), at most 750 new tokens, resampled while any token repeats more than 100
times in a row; the acoustic codebooks are decoded as AcousticDecoding says.
MaxMelLength caps decoded codec frames, including the regenerated final prompt frame. MaxNewSemanticTokens separately limits new T5 tokens; generation respects the smaller budget.
Constructors
| Constructor | Summary |
|---|---|
PhemeOptions() |
Creates the paper’s small model’s options (Budzianowski et al. |
PhemeOptions(PhemeOptions other) |
Creates a copy of other. |
Properties
| Property | Summary |
|---|---|
PhemeAcousticDecoding AcousticDecoding { get; set; } |
Which acoustic codebooks are decoded with MaskGIT and which greedily. |
SpeechTokenizerOptions SpeechTokenizer { get; set; } |
The SpeechTokenizer whose codes Pheme models (its released configuration by default). |
bool UseSpeakerEmbedding { get; set; } |
Whether the acoustic model reads a pyannote speaker embedding (true; false in the released large checkpoint). |
double SpeakerEmbeddingDropout { get; set; } |
Dropout on the speaker embedding before its projection (0.05). |
double Temperature { get; set; } |
Semantic sampling temperature (0.7). |
int AcousticCodebooks { get; set; } |
Acoustic codebooks after the semantic one (7). |
int AcousticConvExpansion { get; set; } |
Acoustic Conformer convolution expansion factor (2). |
int AcousticConvKernel { get; set; } |
Acoustic Conformer depthwise convolution kernel (5). |
int AcousticFeedForwardMultiplier { get; set; } |
Acoustic Conformer feed-forward multiplier (4). |
int AcousticHeadDim { get; set; } |
Acoustic Conformer per-head width (64). |
int AcousticHeads { get; set; } |
Acoustic Conformer attention heads (8). |
int AcousticLayers { get; set; } |
Acoustic Conformer layers (3; 8 in the large model). |
int AcousticTrainingSteps { get; set; } |
Updates after which the acoustic learning rate reaches zero (800,000, §4.1). |
int AcousticWarmupSteps { get; set; } |
Linear warm-up of the acoustic learning rate, in updates (10,000, §4.1). |
int CodebookSize { get; set; } |
Codes per acoustic codebook (1024). |
int FftSize { get; set; } |
Inherited spectrogram metadata; the native codec owns its analysis configuration. |
int HopSize { get; set; } |
The waveform hop, paired with the SpeechTokenizer stride product. |
int MaskGitSteps { get; set; } |
MaskGIT (confidence-based) steps for each codebook AcousticDecoding decodes that way (16). |
int MaxConsecutiveRepeats { get; set; } |
Semantic tokens are resampled while one token repeats more than this many times in a row (100). |
int MaxNewSemanticTokens { get; set; } |
Most semantic tokens generated after the prompt (750). |
int MelChannels { get; set; } |
Inherited mel-output metadata; native Pheme emits codec-decoded waveforms. |
int SampleRate { get; set; } |
The waveform rate, paired with SpeechTokenizer.SampleRate. |
int SamplingSeed { get; set; } |
Seed of the training draws (prompt split, mask ratio, codebook level) and of sampling. |
int SemanticCodes { get; set; } |
Semantic codes (1024). |
int TextFeedForwardDim { get; set; } |
T5 feed-forward width d_ff (2048). |
int TextKeyValueDim { get; set; } |
T5 per-head width d_kv (64). |
int TextModelDim { get; set; } |
T5 model width d_model (512). |
int TextTrainingSteps { get; set; } |
Updates after which the text-to-semantic learning rate reaches zero (800,000, §4.1). |
int TextWarmupSteps { get; set; } |
Linear warm-up of the text-to-semantic learning rate, in updates (10,000, §4.1). |
int TopK { get; set; } |
Semantic top-k sampling (210). |
int VocabSize { get; set; } |
The derived phoneme/special/semantic vocabulary size. |
Methods
| Method | Summary |
|---|---|
PhemeOptions Large() |
The paper’s large model (Table 5): 14 + 14 T5 layers and an 8-layer acoustic Conformer. |
PhemeOptions OfficialLargeCheckpoint() |
The released large checkpoint (PolyAI/pheme, config_s2a.json): the large model trained without the speaker embedding (use_spkr_emb: false). |
PhemeOptions OfficialSmallCheckpoint() |
The released small checkpoint (PolyAI/pheme_small, config_s2a.json): the small model with a 768-wide acoustic Conformer. |