PhemeOptions

Options & Configuration · AiDotNet.TextToSpeech.CodecBased

Options for Pheme (Budzianowski et al. 2024, “Pheme: Efficient and Conversational Speech Generation”).

Inheritance: ModelOptions → TtsModelOptions → PhemeOptions

For Beginners

These options set the sizes and training settings of Pheme’s two networks. The defaults reproduce the released small model.

How It Works

Pheme is a two-stage model over SpeechTokenizer codes: a T5 encoder–decoder turns phonemes into semantic tokens (the first SpeechTokenizer codebook), and a SoundStorm-style Conformer fills in the seven acoustic codebooks non-autoregressively, conditioned on the semantic tokens, a prompt and a speaker embedding.

The defaults are the paper’s small model (Table 5: a 6 + 6-layer T5 of width 512, a 3-layer acoustic Conformer of width 1024) and its training recipe (§4.1: both networks trained with AdamW, β = 0.9 / 0.98, at 5e-4 with 10,000 warm-up steps and linear decay to zero over 800,000 steps; the paper also writes “decayed from 2×10⁻⁴”, which contradicts its stated rate, so the stated 5e-4 is the peak). Large is the paper’s large model. The released checkpoints differ from the paper’s text: OfficialSmallCheckpoint (PolyAI/pheme_small) has a 768-wide acoustic model, and OfficialLargeCheckpoint (PolyAI/pheme) no speaker embedding.

What the paper leaves to its code follows the reference implementation (PolyAI-LDN/pheme): the text-to-semantic model is Hugging Face’s T5ForConditionalGeneration v1.0 (d_ff 2048, d_kv 64, 8 heads, ReLU, dropout 0.1, 32 relative buckets up to distance 128), with the Trainer’s defaults where the paper is silent (no weight decay, gradient norm clipped to 1); the acoustic model is lucidrains’ SoundStorm Conformer (8 heads of 64, feed-forward ×4, convolution expansion 2, kernel 5, dropout 0.1) with a 512 → width projection of the L2-normalized pyannote speaker embedding (dropout 0.05) and PyTorch’s default AdamW weight decay 0.01; codes are SpeechTokenizer’s (1024 semantic and 1024 acoustic codes, 7 acoustic codebooks, 16 kHz, 50 frames per second). Synthesis samples semantic tokens at temperature 0.7 from the top 210 (§4.1), at most 750 new tokens, resampled while any token repeats more than 100 times in a row; the acoustic codebooks are decoded as AcousticDecoding says.

MaxMelLength caps decoded codec frames, including the regenerated final prompt frame. MaxNewSemanticTokens separately limits new T5 tokens; generation respects the smaller budget.

Constructors

Constructor Summary
PhemeOptions() Creates the paper’s small model’s options (Budzianowski et al.
PhemeOptions(PhemeOptions other) Creates a copy of other.

Properties

Property Summary
PhemeAcousticDecoding AcousticDecoding { get; set; } Which acoustic codebooks are decoded with MaskGIT and which greedily.
SpeechTokenizerOptions SpeechTokenizer { get; set; } The SpeechTokenizer whose codes Pheme models (its released configuration by default).
bool UseSpeakerEmbedding { get; set; } Whether the acoustic model reads a pyannote speaker embedding (true; false in the released large checkpoint).
double SpeakerEmbeddingDropout { get; set; } Dropout on the speaker embedding before its projection (0.05).
double Temperature { get; set; } Semantic sampling temperature (0.7).
int AcousticCodebooks { get; set; } Acoustic codebooks after the semantic one (7).
int AcousticConvExpansion { get; set; } Acoustic Conformer convolution expansion factor (2).
int AcousticConvKernel { get; set; } Acoustic Conformer depthwise convolution kernel (5).
int AcousticFeedForwardMultiplier { get; set; } Acoustic Conformer feed-forward multiplier (4).
int AcousticHeadDim { get; set; } Acoustic Conformer per-head width (64).
int AcousticHeads { get; set; } Acoustic Conformer attention heads (8).
int AcousticLayers { get; set; } Acoustic Conformer layers (3; 8 in the large model).
int AcousticTrainingSteps { get; set; } Updates after which the acoustic learning rate reaches zero (800,000, §4.1).
int AcousticWarmupSteps { get; set; } Linear warm-up of the acoustic learning rate, in updates (10,000, §4.1).
int CodebookSize { get; set; } Codes per acoustic codebook (1024).
int FftSize { get; set; } Inherited spectrogram metadata; the native codec owns its analysis configuration.
int HopSize { get; set; } The waveform hop, paired with the SpeechTokenizer stride product.
int MaskGitSteps { get; set; } MaskGIT (confidence-based) steps for each codebook AcousticDecoding decodes that way (16).
int MaxConsecutiveRepeats { get; set; } Semantic tokens are resampled while one token repeats more than this many times in a row (100).
int MaxNewSemanticTokens { get; set; } Most semantic tokens generated after the prompt (750).
int MelChannels { get; set; } Inherited mel-output metadata; native Pheme emits codec-decoded waveforms.
int SampleRate { get; set; } The waveform rate, paired with SpeechTokenizer.SampleRate.
int SamplingSeed { get; set; } Seed of the training draws (prompt split, mask ratio, codebook level) and of sampling.
int SemanticCodes { get; set; } Semantic codes (1024).
int TextFeedForwardDim { get; set; } T5 feed-forward width d_ff (2048).
int TextKeyValueDim { get; set; } T5 per-head width d_kv (64).
int TextModelDim { get; set; } T5 model width d_model (512).
int TextTrainingSteps { get; set; } Updates after which the text-to-semantic learning rate reaches zero (800,000, §4.1).
int TextWarmupSteps { get; set; } Linear warm-up of the text-to-semantic learning rate, in updates (10,000, §4.1).
int TopK { get; set; } Semantic top-k sampling (210).
int VocabSize { get; set; } The derived phoneme/special/semantic vocabulary size.

Methods

Method Summary
PhemeOptions Large() The paper’s large model (Table 5): 14 + 14 T5 layers and an 8-layer acoustic Conformer.
PhemeOptions OfficialLargeCheckpoint() The released large checkpoint (PolyAI/pheme, config_s2a.json): the large model trained without the speaker embedding (use_spkr_emb: false).
PhemeOptions OfficialSmallCheckpoint() The released small checkpoint (PolyAI/pheme_small, config_s2a.json): the small model with a 768-wide acoustic Conformer.