Pheme<T>
Models & Types · AiDotNet.TextToSpeech.CodecBased
Pheme: efficient and conversational speech generation in two stages over SpeechTokenizer codes, a T5 text-to-semantic model and a non-autoregressive SoundStorm-style semantic-to-acoustic model.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → Pheme<T>
Implements: ITtsModel<T>
For Beginners
Pheme first writes down “what is said” as a short sequence of speech tokens, then fills in “how it sounds” for a given voice, and a codec turns those tokens into audio.
How It Works
References: “Pheme: Efficient and Conversational Speech Generation” (Budzianowski et al., PolyAI, 2024) and its reference implementation (PolyAI-LDN/pheme) for what the paper leaves unstated.
Text to semantic (T2S): text becomes espeak-style phonemes (EnglishG2P), read by a T5
encoder–decoder (T5ForConditionalGeneration v1.0) over one vocabulary of the phoneme symbols and the 1024
semantic codes: the input is <bos> spkr_1 phonemes <eos>, the target <bos> spkr_1 semantic tokens <eos>, trained with the teacher-forced cross-entropy.
Semantic to acoustic (S2A, after SoundStorm, Borsos et al. 2023): each of the seven acoustic codebooks has its own embedding; a training step picks one codebook level, keeps a random prompt prefix, masks a cosine-scheduled fraction of the level’s remaining tokens, sums the embeddings of the lower levels, the partly masked level and the semantic tokens, adds the projected L2-normalized pyannote speaker embedding, runs a SoundStorm Conformer and predicts the masked tokens with the level’s head (cross-entropy over the masked positions).
Synthesis (reference transformer_infer.py): the voice prompt’s transcript precedes the text and its
semantic tokens start the T5 decoder, which samples the continuation (temperature 0.7, top 210, resampled while a
token repeats more than 100 times in a row); the acoustic model fills the first codebook with 16 MaskGIT steps and
the other six with one greedy step each, after the prompt’s acoustic tokens (the reference code’s order; the
paper’s text describes the reverse, AcousticDecoding); SpeechTokenizer decodes the codes.
Two reference behaviours are kept as written: the speaker-embedding dropout is applied at inference too (the code
calls F.dropout without the training flag), and the last prompt frame is regenerated and returned with the
synthesized frames (the prompt split counts the leading speaker token).
One is not: the reference’s MaskGIT loop passes the initial, fully masked sequence to the model at every step
(self.forward(inputs, ...), never the updated state.cur_seqs), so its later steps refine nothing; here
each step reads the tokens decoded so far, as MaskGIT (Chang et al. 2022) and SoundStorm decode and the paper
describes.
Training data: Tokens are the phoneme ids (see
EncodePhonemes), CodecTokens the SpeechTokenizer codes
[frames, 8] (or Audio, which the model encodes), and with the speaker
embedding the recording it is computed from (SpeakerReference or the audio).
CurrentStage selects the network a step trains.
Example
var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional,
NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var pheme = new Pheme<float>(architecture, new PhemeOptions());
pheme.Voice = pheme.CreateVoice(promptAudio16kHz, "the prompt's transcript");
var audio = pheme.Synthesize("Hello there.");
Constructors
| Constructor | Summary |
|---|---|
Pheme(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
Pheme(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Calls the fully specified overload with the omitted parameters’ defaults. |
Pheme(NeuralNetworkArchitecture<T> architecture, PhemeOptions options) |
Calls the fully specified overload with the omitted parameters’ defaults. |
Pheme(NeuralNetworkArchitecture<T> architecture, PhemeOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a native, trainable Pheme. |
Pheme(NeuralNetworkArchitecture<T> architecture, string modelPath) |
Calls the fully specified overload with the omitted parameters’ defaults. |
Pheme(NeuralNetworkArchitecture<T> architecture, string modelPath, PhemeOptions options) |
Creates a native, trainable Pheme. |
Properties
| Property | Summary |
|---|---|
PhemeTrainingStage CurrentStage { get; set; } |
The network a training step updates. |
SpeechTokenizer<T> Codec { get; } |
The SpeechTokenizer whose codes Pheme models. |
int CodecTokenCodebooks { get; } |
|
int CodecTokenVocabulary { get; } |
|
int MaxTextLength { get; } |
|
int SemanticToAcousticUpdates { get; set; } |
Training steps taken by the acoustic network. |
int TextToSemanticUpdates { get; set; } |
Training steps taken by each network (they set the learning-rate schedules). |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
TtsVoice<T> CreateVoice(Tensor<T> recording, string transcript) |
A voice for Voice: a 16 kHz prompt recording and its transcript. |
int[] EncodePhonemes(IEnumerable<string> symbols) |
Pheme’s token id of each phoneme symbol (symbols outside its table are dropped, as the reference drops them), without the <bos> spkr_1 … <eos> frame. |
void LoadSemanticToAcousticWeights(string path) |
Loads the released acoustic model: the state_dict of s2a.ckpt (its model.* tensors) saved as safetensors or .pt; the Lightning checkpoint itself also holds optimizer state and is not read directly. |
void LoadSpeakerEncoderWeights(string path) |
Loads the pyannote/embedding speaker model’s state dictionary (safetensors or .pt). |
void LoadTextToSemanticWeights(string path) |
Loads the released text-to-semantic model: the state dictionary of T5ForConditionalGeneration as safetensors, or the released t2s.bin (a PyTorch archive) saved with a .pt extension. |