AudioPaLM<T>
Models & Types · AiDotNet.TextToSpeech.MultiModal
AudioPaLM (Rubenstein et al. 2023): a text-pretrained decoder-only language model whose vocabulary is extended by audio tokens, so one model reads and writes both text and speech — text to speech, speech recognition, speech translation to text and speech-to-speech translation in the speaker’s voice.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → AudioPaLM<T>
For Beginners
AudioPaLM is a chatbot-style language model that has learned “audio words” alongside ordinary words. Give it text and a task, and it writes audio words that a separate decoder turns into speech; give it speech, and it can write the transcript or a translation.
How It Works
Speech becomes audio tokens through a speech encoder’s k-means codes (§3.1: w2v-BERT, 1024 tokens at 25 Hz);
they follow the text tokens in one vocabulary, the embedding table (shared with the output layer) grown by their rows,
initialized to zero (§3.2). A task is a text tag such as [TTS English] or [S2ST English French]
before the input (§3.4); the model continues with the output and an end token, and is trained with the loss on the
output only. Audio tokens become speech through SoundStorm or AudioLM’s acoustic stages, continuing a 3-second voice
prompt (§3.3).
The decoder is PaLM’s (Chowdhery et al. 2022 §2) — PaLM-2’s architecture is not published — or the public
Gemma’s (Gemma2B), whose pretrained weights stand in for the text checkpoint the paper
starts from.
Example
var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var options = AudioPaLMOptions.Gemma2B();
options.SentencePieceModelPath = "gemma-2b/tokenizer.model";
var model = new AudioPaLM<float>(architecture, options);
model.LoadGemmaWeights("gemma-2b");
model.FitSpeechTokenizer(recordings); // the audio tokens' k-means codebook
model.Train(new TtsTrainingSample<float> { Tokens = textIds, Audio = recording });
model.Voice = new TtsVoice<float> { ReferenceAudio = voiceRecording };
var audio = model.Synthesize("Hello world.");
string text = model.Transcribe(recording, "English");
Constructors
| Constructor | Summary |
|---|---|
AudioPaLM(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
AudioPaLM(NeuralNetworkArchitecture<T> architecture, AudioPaLMOptions options) |
Creates a native, trainable AudioPaLM. |
AudioPaLM(NeuralNetworkArchitecture<T> architecture, string modelPath, AudioPaLMOptions options) |
Creates an ONNX-backed AudioPaLM for inference. |
Properties
| Property | Summary |
|---|---|
AudioLM<T> AudioLMDetokenizer { get; } |
The AudioLM whose acoustic stages render audio tokens (with the AudioLM detokenizer). |
SoundStorm<T> SoundStormDetokenizer { get; } |
The SoundStorm acoustic model that renders audio tokens (with the SoundStorm detokenizer). |
W2vBert<T> SpeechEncoder { get; } |
The w2v-BERT model whose k-means codes are the audio tokens (with the w2v-BERT tokenizer). |
bool TrainsOnSampleObjective { get; } |
|
int AudioOffset { get; } |
The first audio token’s id: the text vocabulary’s size (§3.2). |
int MaxTextLength { get; } |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
T TrainTask(string tag, IReadOnlyList<int> input, IReadOnlyList<int> output) |
One update of any task (§3.4): tag (such as [ASR French] or [ASR AST S2ST English French]) and input in, output and the end token out, all as decoder ids (text ids, or audio codes offset by AudioOffset). |
Tensor<T> Detokenize(IReadOnlyList<int> codes, Tensor<T> voiceRecording) |
Renders audio token codes to a 16 kHz waveform with the detokenizer, continuing the voice prompt taken from voiceRecording (§3.3). |
Tensor<T> Synthesize(string text) |
|
Tensor<T> TranslateSpeech(Tensor<T> recording, string sourceLanguage, string targetLanguage) |
Speech-to-speech translation ([S2ST source target]): the translated speech in the voice of the source recording, whose first seconds are the voice prompt (§3.3). |
int[] AudioTokensOf(Tensor<T> recording) |
The audio tokens of a 16 kHz recording (codes in [0, AudioTokens)). |
int[] Generate(IReadOnlyList<int> prefix, bool audioOutput) |
Continues prefix (decoder ids) until the end token or MaxNewTokens, choosing among audio tokens (audioOutput) or text tokens, and the end token after the first; greedy, or sampled at Temperature. |
int[] TaskPrefix(string tag, IReadOnlyList<int> input) |
The decoder ids of a task: the start token, the tag’s text tokens, then the input’s ids (§3.4). |
int[] TokenizeText(string text) |
The text token ids of text (the decoder’s SentencePiece model). |
string DecodeText(IEnumerable<int> ids) |
The text of text token ids (audio ids and the start and end tokens dropped). |
string Transcribe(Tensor<T> recording, string language) |
Speech recognition ([ASR language], the language’s English name): the transcript of a 16 kHz recording. |
string TranslateSpeechToText(Tensor<T> recording, string sourceLanguage, string targetLanguage) |
Speech translation to text ([AST source target]). |
void FitSpeechTokenizer(IEnumerable<Tensor<T>> recordings) |
Fits the audio tokens’ k-means codebook on 16 kHz recordings: w2v-BERT’s semantic tokenizer on its embeddings as they are (§3.1), or k-means over w2v-BERT 2.0’s W2vBert2Layer embeddings. |
void LoadGemmaWeights(string directory) |
Loads a Hugging Face Gemma checkpoint directory into the decoder (its text rows; the audio rows zeroed, §3.2) and, unless a tokenizer is set, its tokenizer.model. |
void LoadPalmPytorchWeights(string path) |
Loads a lucidrains/PaLM-pytorch state dict into the PaLM decoder (its text rows; the audio rows zeroed). |
void LoadSpeechTokenizerCodebook(double[] centroids) |
Sets the w2v-BERT 2.0 tokenizer’s k-means centroids [AudioTokens, 1024]. |
void LoadW2vBert2Weights(string path) |
Loads w2v-BERT 2.0 (facebook/w2v-bert-2.0) for the w2v-BERT 2.0 tokenizer. |