SoundStormOptions
Options & Configuration · AiDotNet.TextToSpeech.CodecBased
Options for SoundStorm (Borsos et al. 2023, “SoundStorm: Efficient Parallel Audio Generation”).
Inheritance: ModelOptions → TtsModelOptions → SoundStormOptions
For Beginners
SoundStorm turns “what is said” tokens into the audio codec’s tokens all at once instead of one by one, which makes it about a hundred times faster than generating them in order.
How It Works
The defaults are the paper’s acoustic generator (§4.1): a 350M-parameter bidirectional Conformer (12 layers, 16 heads, width 1024, feed-forward 4096, convolution kernel 5, rotary positions) over a SoundStream codec at 50 frames per second with Q = 12 RVQ levels of 1024 codes, conditioned on semantic tokens (1024 k-means clusters at 25 Hz, each duplicated to the codec’s 50 Hz). Decoding runs (16, 1, …, 1) iterations: 16 confidence-based iterations on the first level, whose last iteration is greedy (§3.3), then one greedy pass per finer level.
Synthesis from text uses the paper’s text-to-semantic model (§5): ByT5-large (36 encoder and 12 decoder layers,
width 1536, feed-forward 3840, Xue et al. 2022), sampling semantic tokens at temperature 0.9 from the top 125. Only
its decoder trains; the encoder comes from a text-pretrained ByT5 checkpoint (LoadTextEncoderWeights).
The paper gives no optimizer, dropout or convolution expansion, and no released code exists. Those gaps follow
lucidrains/soundstorm-pytorch: no dropout, convolution expansion 2, a learned mask embedding, and its trainer’s
Adam (β = 0.9, 0.99, ε = 1e-8, no weight decay) at 3e-4 after a linear warmup from 1e-5, decayed by a cosine over
the training steps, with gradients clipped at norm 0.5. Its trainer requires the warmup and total steps and gives
neither a default, so WarmupSteps and TrainingSteps are left for you to set; while they
are 0 the learning rate stays at LearningRate.
Constructors
| Constructor | Summary |
|---|---|
SoundStormOptions() |
Creates the paper’s options (§4.1, §5). |
SoundStormOptions(SoundStormOptions other) |
Creates a copy of other. |
Properties
| Property | Summary |
|---|---|
SoundStreamOptions SoundStream { get; set; } |
The SoundStream codec’s options. |
bool IncludeTextToSemantic { get; set; } |
Whether the §5 text-to-semantic model (ByT5) is built (true). |
double MaxGradientNorm { get; set; } |
Gradient norm clip (0.5). |
double Temperature { get; set; } |
Sampling temperature for semantic tokens (0.9). |
double TextDropoutRate { get; set; } |
ByT5’s dropout (0.1, the released checkpoints’ dropout_rate). |
double WarmupStartLearningRate { get; set; } |
Learning rate at the start of the warmup (1e-5). |
int CodebookSize { get; set; } |
C, the codes per RVQ level (1024). |
int ConformerHeadDim { get; set; } |
Width of each attention head (64: 16 heads over width 1024). |
int ConformerLayers { get; set; } |
Conformer layers (12). |
int ConvExpansion { get; set; } |
Expansion of the convolution module (2, from lucidrains/soundstorm-pytorch; the paper does not say). |
int ConvKernel { get; set; } |
Convolution kernel size (5). |
int FeedForwardMultiplier { get; set; } |
Feed-forward width as a multiple of the model width (4: 4096 over 1024). |
int FirstLevelIterations { get; set; } |
Confidence-based iterations on the first RVQ level (16); every finer level takes one greedy pass. |
int HopSize { get; set; } |
The codec hop; must match the product of SoundStream.Ratios. |
int MaxNewSemanticTokens { get; set; } |
The most semantic tokens generated (750: 30 seconds at 25 Hz, §5). |
int NumCodebooks { get; set; } |
Q, the RVQ levels modeled (12). |
int SampleRate { get; set; } |
The codec sample rate; must match SoundStream.SampleRate. |
int SamplingSeed { get; set; } |
Seed of synthesis’s and training’s random draws. |
int SemanticCodes { get; set; } |
Semantic token vocabulary (1024 w2v-BERT k-means clusters). |
int SemanticFrameRate { get; set; } |
Semantic tokens per second (25); each is duplicated to the codec’s frame rate. |
int TextFeedForwardDim { get; set; } |
ByT5’s feed-forward width (3840). |
int TextKeyValueDim { get; set; } |
ByT5’s attention head width (64). |
int TextModelDim { get; set; } |
ByT5’s width (1536). |
int TopK { get; set; } |
Semantic tokens are sampled from the top K (125). |
int TrainingSteps { get; set; } |
Total training steps the cosine decay spans. |
int VocabSize { get; set; } |
The derived ByT5 byte and semantic token vocabulary. |
int WarmupSteps { get; set; } |
Linear warmup steps. |
Methods
| Method | Summary |
|---|---|
SoundStormOptions DialogueSynthesis(int[] codecStrides) |
The paper’s dialogue system (§5): semantic tokens from 4096 k-means clusters of a 0.6B BEST-RQ at 25 Hz, over a 24 kHz SoundStream at 50 frames per second with 12 RVQ levels. |