AudioLMOptions
Options & Configuration · AiDotNet.Audio.Generation
Options for AudioLM (Borsos et al. 2022, “AudioLM: a Language Modeling Approach to Audio Generation”).
Inheritance: ModelOptions → AudioLMOptions
For Beginners
AudioLM continues a recording: given a few seconds of speech or piano, it writes what comes next, keeping the voice and the recording conditions. It has no text input; SPEAR-TTS adds one.
How It Works
The defaults are the paper’s speech model (§IV-B): semantic tokens are w2v-BERT XL’s 7th masked-prediction layer clustered into K = 1024 (25 a second); acoustic tokens are a 16 kHz SoundStream with strides (2, 4, 5, 8) and 12 RVQ levels of 1024 codes (50 frames a second), split into Q′ = 4 coarse and 8 fine levels. Each stage is a decoder-only Transformer with 12 layers, 16 heads, width 1024, feed-forward 4096, dropout 0.1 and T5-style relative positions, trained on random crops of 30, 10 and 3 seconds; sampling uses temperatures 0.6, 0.8 and 0.6.
The paper states no optimizer or learning rate (16 TPUv4s, batch 256, 1M steps per stage) and released no code. Those gaps follow lucidrains/audiolm-pytorch: Adam (β = 0.9 / 0.99, ε = 1e-8, no weight decay) at a constant 3e-4 with gradients clipped at norm 0.5, the gated-GELU (GEGLU) feed-forward and untied output heads.
Constructors
| Constructor | Summary |
|---|---|
AudioLMOptions() |
Creates the paper’s options. |
AudioLMOptions(AudioLMOptions other) |
Creates a copy of other. |
Properties
| Property | Summary |
|---|---|
SoundStreamOptions SoundStream { get; set; } |
The SoundStream codec: 16 kHz, strides (2, 4, 5, 8), 12 levels of 1024 codes. |
double CoarseCropSeconds { get; set; } |
Training crop of the coarse stage, in seconds (10). |
double CoarseTemperature { get; set; } |
Sampling temperature of the coarse stage (0.8). |
double DropoutRate { get; set; } |
Dropout (0.1). |
double FineCropSeconds { get; set; } |
Training crop and generation chunk of the fine stage, in seconds (3). |
double FineTemperature { get; set; } |
Sampling temperature of the fine stage (0.6). |
double LearningRate { get; set; } |
Learning rate (3e-4, lucidrains/audiolm-pytorch; constant). |
double MaxGradientNorm { get; set; } |
Gradient norm clip (0.5, lucidrains/audiolm-pytorch). |
double SemanticCropSeconds { get; set; } |
Training crop of the semantic stage, in seconds (30). |
double SemanticTemperature { get; set; } |
Sampling temperature of the semantic stage (0.6). |
int CoarseCodebooks { get; set; } |
Q′, the levels the coarse stage models (4). |
int CodebookSize { get; set; } |
N, codes per level (1024). |
int FeedForwardDim { get; set; } |
Feed-forward width (4096). |
int KeyValueDim { get; set; } |
Width of each head (64: 16 heads over 1024). |
int ModelDim { get; set; } |
Transformer width (1024). |
int NumCodebooks { get; set; } |
Q, SoundStream’s RVQ levels (12). |
int NumHeads { get; set; } |
Attention heads (16). |
int NumLayers { get; set; } |
Layers per stage (12). |
int SampleRate { get; set; } |
Audio sample rate (16 kHz). |
int SemanticCodes { get; set; } |
Semantic token vocabulary (K = 1024). |
int SemanticFrameRate { get; set; } |
Semantic tokens per second (25). |