Pheme<T>

Models & Types · AiDotNet.TextToSpeech.CodecBased

Pheme: efficient and conversational speech generation in two stages over SpeechTokenizer codes, a T5 text-to-semantic model and a non-autoregressive SoundStorm-style semantic-to-acoustic model.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → Pheme<T>

Implements: ITtsModel<T>

For Beginners

Pheme first writes down “what is said” as a short sequence of speech tokens, then fills in “how it sounds” for a given voice, and a codec turns those tokens into audio.

How It Works

References: “Pheme: Efficient and Conversational Speech Generation” (Budzianowski et al., PolyAI, 2024) and its reference implementation (PolyAI-LDN/pheme) for what the paper leaves unstated.

Text to semantic (T2S): text becomes espeak-style phonemes (EnglishG2P), read by a T5 encoder–decoder (T5ForConditionalGeneration v1.0) over one vocabulary of the phoneme symbols and the 1024 semantic codes: the input is <bos> spkr_1 phonemes <eos>, the target <bos> spkr_1 semantic tokens <eos>, trained with the teacher-forced cross-entropy.

Semantic to acoustic (S2A, after SoundStorm, Borsos et al. 2023): each of the seven acoustic codebooks has its own embedding; a training step picks one codebook level, keeps a random prompt prefix, masks a cosine-scheduled fraction of the level’s remaining tokens, sums the embeddings of the lower levels, the partly masked level and the semantic tokens, adds the projected L2-normalized pyannote speaker embedding, runs a SoundStorm Conformer and predicts the masked tokens with the level’s head (cross-entropy over the masked positions).

Synthesis (reference transformer_infer.py): the voice prompt’s transcript precedes the text and its semantic tokens start the T5 decoder, which samples the continuation (temperature 0.7, top 210, resampled while a token repeats more than 100 times in a row); the acoustic model fills the first codebook with 16 MaskGIT steps and the other six with one greedy step each, after the prompt’s acoustic tokens (the reference code’s order; the paper’s text describes the reverse, AcousticDecoding); SpeechTokenizer decodes the codes. Two reference behaviours are kept as written: the speaker-embedding dropout is applied at inference too (the code calls F.dropout without the training flag), and the last prompt frame is regenerated and returned with the synthesized frames (the prompt split counts the leading speaker token). One is not: the reference’s MaskGIT loop passes the initial, fully masked sequence to the model at every step (self.forward(inputs, ...), never the updated state.cur_seqs), so its later steps refine nothing; here each step reads the tokens decoded so far, as MaskGIT (Chang et al. 2022) and SoundStorm decode and the paper describes.

Training data: Tokens are the phoneme ids (see EncodePhonemes), CodecTokens the SpeechTokenizer codes [frames, 8] (or Audio, which the model encodes), and with the speaker embedding the recording it is computed from (SpeakerReference or the audio). CurrentStage selects the network a step trains.

Example

var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional,
    NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var pheme = new Pheme<float>(architecture, new PhemeOptions());
pheme.Voice = pheme.CreateVoice(promptAudio16kHz, "the prompt's transcript");
var audio = pheme.Synthesize("Hello there.");

Constructors

Constructor Summary
Pheme(NeuralNetworkArchitecture<T> architecture) Calls the fully specified overload with the omitted parameters’ defaults.
Pheme(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Calls the fully specified overload with the omitted parameters’ defaults.
Pheme(NeuralNetworkArchitecture<T> architecture, PhemeOptions options) Calls the fully specified overload with the omitted parameters’ defaults.
Pheme(NeuralNetworkArchitecture<T> architecture, PhemeOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a native, trainable Pheme.
Pheme(NeuralNetworkArchitecture<T> architecture, string modelPath) Calls the fully specified overload with the omitted parameters’ defaults.
Pheme(NeuralNetworkArchitecture<T> architecture, string modelPath, PhemeOptions options) Creates a native, trainable Pheme.

Properties

Property Summary
PhemeTrainingStage CurrentStage { get; set; } The network a training step updates.
SpeechTokenizer<T> Codec { get; } The SpeechTokenizer whose codes Pheme models.
int CodecTokenCodebooks { get; }
int CodecTokenVocabulary { get; }
int MaxTextLength { get; }
int SemanticToAcousticUpdates { get; set; } Training steps taken by the acoustic network.
int TextToSemanticUpdates { get; set; } Training steps taken by each network (they set the learning-rate schedules).

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
TtsVoice<T> CreateVoice(Tensor<T> recording, string transcript) A voice for Voice: a 16 kHz prompt recording and its transcript.
int[] EncodePhonemes(IEnumerable<string> symbols) Pheme’s token id of each phoneme symbol (symbols outside its table are dropped, as the reference drops them), without the <bos> spkr_1 … <eos> frame.
void LoadSemanticToAcousticWeights(string path) Loads the released acoustic model: the state_dict of s2a.ckpt (its model.* tensors) saved as safetensors or .pt; the Lightning checkpoint itself also holds optimizer state and is not read directly.
void LoadSpeakerEncoderWeights(string path) Loads the pyannote/embedding speaker model’s state dictionary (safetensors or .pt).
void LoadTextToSemanticWeights(string path) Loads the released text-to-semantic model: the state dictionary of T5ForConditionalGeneration as safetensors, or the released t2s.bin (a PyTorch archive) saved with a .pt extension.