AudioLMOptions

Options & Configuration · AiDotNet.Audio.Generation

Options for AudioLM (Borsos et al. 2022, “AudioLM: a Language Modeling Approach to Audio Generation”).

Inheritance: ModelOptions → AudioLMOptions

For Beginners

AudioLM continues a recording: given a few seconds of speech or piano, it writes what comes next, keeping the voice and the recording conditions. It has no text input; SPEAR-TTS adds one.

How It Works

The defaults are the paper’s speech model (§IV-B): semantic tokens are w2v-BERT XL’s 7th masked-prediction layer clustered into K = 1024 (25 a second); acoustic tokens are a 16 kHz SoundStream with strides (2, 4, 5, 8) and 12 RVQ levels of 1024 codes (50 frames a second), split into Q′ = 4 coarse and 8 fine levels. Each stage is a decoder-only Transformer with 12 layers, 16 heads, width 1024, feed-forward 4096, dropout 0.1 and T5-style relative positions, trained on random crops of 30, 10 and 3 seconds; sampling uses temperatures 0.6, 0.8 and 0.6.

The paper states no optimizer or learning rate (16 TPUv4s, batch 256, 1M steps per stage) and released no code. Those gaps follow lucidrains/audiolm-pytorch: Adam (β = 0.9 / 0.99, ε = 1e-8, no weight decay) at a constant 3e-4 with gradients clipped at norm 0.5, the gated-GELU (GEGLU) feed-forward and untied output heads.

Constructors

Constructor Summary
AudioLMOptions() Creates the paper’s options.
AudioLMOptions(AudioLMOptions other) Creates a copy of other.

Properties

Property Summary
SoundStreamOptions SoundStream { get; set; } The SoundStream codec: 16 kHz, strides (2, 4, 5, 8), 12 levels of 1024 codes.
double CoarseCropSeconds { get; set; } Training crop of the coarse stage, in seconds (10).
double CoarseTemperature { get; set; } Sampling temperature of the coarse stage (0.8).
double DropoutRate { get; set; } Dropout (0.1).
double FineCropSeconds { get; set; } Training crop and generation chunk of the fine stage, in seconds (3).
double FineTemperature { get; set; } Sampling temperature of the fine stage (0.6).
double LearningRate { get; set; } Learning rate (3e-4, lucidrains/audiolm-pytorch; constant).
double MaxGradientNorm { get; set; } Gradient norm clip (0.5, lucidrains/audiolm-pytorch).
double SemanticCropSeconds { get; set; } Training crop of the semantic stage, in seconds (30).
double SemanticTemperature { get; set; } Sampling temperature of the semantic stage (0.6).
int CoarseCodebooks { get; set; } Q′, the levels the coarse stage models (4).
int CodebookSize { get; set; } N, codes per level (1024).
int FeedForwardDim { get; set; } Feed-forward width (4096).
int KeyValueDim { get; set; } Width of each head (64: 16 heads over 1024).
int ModelDim { get; set; } Transformer width (1024).
int NumCodebooks { get; set; } Q, SoundStream’s RVQ levels (12).
int NumHeads { get; set; } Attention heads (16).
int NumLayers { get; set; } Layers per stage (12).
int SampleRate { get; set; } Audio sample rate (16 kHz).
int SemanticCodes { get; set; } Semantic token vocabulary (K = 1024).
int SemanticFrameRate { get; set; } Semantic tokens per second (25).