AudioPaLMOptions

Options & Configuration · AiDotNet.TextToSpeech.MultiModal

Options for AudioPaLM.

Inheritance: ModelOptions → AudioPaLMOptions

For Beginners

AudioPaLM is a text language model that has also learned “audio words”, so the same model can read text aloud, transcribe speech or translate speech into another language’s speech. These settings size the language model, choose how speech becomes audio words and back, and control training and generation.

How It Works

The defaults are PaLM 8B (Chowdhery et al. 2022 Table 1: 32 layers, d 4096, 16 heads of 256, multi-query, d_ff 4·d, a 256k SentencePiece vocabulary) extended by 1024 audio tokens (§3.2), w2v-BERT tokens and SoundStorm, fine-tuned as PaLM §6.1.2 fine-tunes (Adafactor at a constant 5e-5, dropout 0.1, loss masked on the inputs). Gemma2B configures the public Gemma-2B decoder instead.

Constructors

Constructor Summary
AudioPaLMOptions() Creates PaLM 8B’s configuration.

Properties

Property Summary
AudioLMOptions AudioLM { get; set; } AudioLM’s options (its acoustic stages read AudioPaLM’s tokens).
AudioPalmDetokenizer Detokenizer { get; set; } How audio tokens become speech (SoundStorm).
AudioPalmSpeechTokenizer SpeechTokenizer { get; set; } How speech becomes audio tokens (w2v-BERT).
OnnxModelOptions OnnxOptions { get; set; } Settings for the exported ONNX graph runtime.
PalmDecoderKind Decoder { get; set; } The decoder’s architecture (PaLM).
SoundStormOptions SoundStorm { get; set; } SoundStorm’s options (its acoustic model only; AudioPaLM supplies the semantic tokens).
W2vBertOptions W2vBert { get; set; } The w2v-BERT tokenizer’s options (w2v-BERT XL, its 7th masked block, 1024 clusters, not normalized).
double Beta1 { get; set; } Adafactor’s first-moment decay (0.9, PaLM §5).
double DropoutRate { get; set; } The language decoder’s training dropout.
double LearningRate { get; set; } The Adafactor learning rate.
double MaxGradientNorm { get; set; } Global gradient-norm clip (1.0, PaLM §5).
double NormEpsilon { get; set; } The norms’ ε (1e-5, PaLM-pytorch’s LayerNorm; Gemma’s RMSNorm uses 1e-6).
double RampFinalLearningRate { get; set; } §5.4.8’s final rate (1e-5).
double RampPeakLearningRate { get; set; } §5.4.8’s peak rate (1e-4).
double RopeTheta { get; set; } The rotary base (10000).
double Temperature { get; set; } Sampling temperature; 0 decodes greedily (0).
double VoicePromptSeconds { get; set; } Seconds of voice prompt the detokenizer continues (3, repeated when the recording is shorter; §3.3).
double ZLossWeight { get; set; } Weight of PaLM’s auxiliary z-loss, 1e-4 · log²Z (§5).
int AudioTokens { get; set; } Audio tokens appended after the text vocabulary (1024, §3.1).
int BeginId { get; set; } The token every sequence starts with (0, T5X’s decoder start; Gemma’s 2).
int EndId { get; set; } The end token (1).
int HeadDim { get; set; } Width of each head (256).
int HiddenDim { get; set; } The language decoder’s hidden width (4096 for PaLM 8B).
int IntermediateSize { get; set; } The MLP’s inner width (16384 = 4·d).
int KeyValueHeads { get; set; } Key/value heads (1: multi-query).
int Layers { get; set; } Transformer layers (32).
int MaxNewTokens { get; set; } Most tokens a generation adds (1500, a minute of 25 Hz audio tokens).
int MaxTextLength { get; set; } The text sequence limit used by generation.
int NumHeads { get; set; } The language decoder’s query attention heads.
int RampDecaySteps { get; set; } Steps over which §5.4.8’s schedule decays exponentially to RampFinalLearningRate.
int RampWarmupSteps { get; set; } Warm-up steps of §5.4.8’s schedule (a linear ramp to RampPeakLearningRate); 0 with RampDecaySteps 0 keeps §3.6’s constant rate.
int SampleRate { get; set; } The speech tokenizer’s waveform sample rate (16 kHz).
int SamplingSeed { get; set; } Seed of sampling and of the detokenizer’s draws.
int TrainingSeed { get; set; } Seed of the training draws (dropout).
int VocabSize { get; set; } The text vocabulary size before the added audio tokens.
int W2vBert2Layer { get; set; } The w2v-BERT 2.0 hidden state clustered (12 of 24, the middle layer, as §3.1 takes USM’s).
string Language { get; set; } The language the TTS task names (English).
string ModelPath { get; set; } The exported ONNX graph path, when a graph is loaded.
string SentencePieceModelPath { get; set; } The text SentencePiece model (PaLM’s is not public; Gemma’s tokenizer.model with Gemma2B).

Methods

Method Summary
AudioPaLMOptions Gemma2B() The public Gemma-2B decoder (Gemma Team 2024: 18 layers, d 2048, 8 query heads and one key/value head of 256, GeGLU d_ff 16384, its own 256k vocabulary with 2 and 1), which LoadGemmaWeights reads.
AudioPaLMOptions PaLM2Ablation(int warmupSteps, int decaySteps) §5.4.8’s optimization for the PaLM-2 experiments: dropout 0.2 and a linear ramp to 1e-4 followed by an exponential decay to 1e-5 over the given steps (the paper does not state them).