SoundStormOptions

Options & Configuration · AiDotNet.TextToSpeech.CodecBased

Options for SoundStorm (Borsos et al. 2023, “SoundStorm: Efficient Parallel Audio Generation”).

Inheritance: ModelOptions → TtsModelOptions → SoundStormOptions

For Beginners

SoundStorm turns “what is said” tokens into the audio codec’s tokens all at once instead of one by one, which makes it about a hundred times faster than generating them in order.

How It Works

The defaults are the paper’s acoustic generator (§4.1): a 350M-parameter bidirectional Conformer (12 layers, 16 heads, width 1024, feed-forward 4096, convolution kernel 5, rotary positions) over a SoundStream codec at 50 frames per second with Q = 12 RVQ levels of 1024 codes, conditioned on semantic tokens (1024 k-means clusters at 25 Hz, each duplicated to the codec’s 50 Hz). Decoding runs (16, 1, …, 1) iterations: 16 confidence-based iterations on the first level, whose last iteration is greedy (§3.3), then one greedy pass per finer level.

Synthesis from text uses the paper’s text-to-semantic model (§5): ByT5-large (36 encoder and 12 decoder layers, width 1536, feed-forward 3840, Xue et al. 2022), sampling semantic tokens at temperature 0.9 from the top 125. Only its decoder trains; the encoder comes from a text-pretrained ByT5 checkpoint (LoadTextEncoderWeights).

The paper gives no optimizer, dropout or convolution expansion, and no released code exists. Those gaps follow lucidrains/soundstorm-pytorch: no dropout, convolution expansion 2, a learned mask embedding, and its trainer’s Adam (β = 0.9, 0.99, ε = 1e-8, no weight decay) at 3e-4 after a linear warmup from 1e-5, decayed by a cosine over the training steps, with gradients clipped at norm 0.5. Its trainer requires the warmup and total steps and gives neither a default, so WarmupSteps and TrainingSteps are left for you to set; while they are 0 the learning rate stays at LearningRate.

Constructors

Constructor Summary
SoundStormOptions() Creates the paper’s options (§4.1, §5).
SoundStormOptions(SoundStormOptions other) Creates a copy of other.

Properties

Property Summary
SoundStreamOptions SoundStream { get; set; } The SoundStream codec’s options.
bool IncludeTextToSemantic { get; set; } Whether the §5 text-to-semantic model (ByT5) is built (true).
double MaxGradientNorm { get; set; } Gradient norm clip (0.5).
double Temperature { get; set; } Sampling temperature for semantic tokens (0.9).
double TextDropoutRate { get; set; } ByT5’s dropout (0.1, the released checkpoints’ dropout_rate).
double WarmupStartLearningRate { get; set; } Learning rate at the start of the warmup (1e-5).
int CodebookSize { get; set; } C, the codes per RVQ level (1024).
int ConformerHeadDim { get; set; } Width of each attention head (64: 16 heads over width 1024).
int ConformerLayers { get; set; } Conformer layers (12).
int ConvExpansion { get; set; } Expansion of the convolution module (2, from lucidrains/soundstorm-pytorch; the paper does not say).
int ConvKernel { get; set; } Convolution kernel size (5).
int FeedForwardMultiplier { get; set; } Feed-forward width as a multiple of the model width (4: 4096 over 1024).
int FirstLevelIterations { get; set; } Confidence-based iterations on the first RVQ level (16); every finer level takes one greedy pass.
int HopSize { get; set; } The codec hop; must match the product of SoundStream.Ratios.
int MaxNewSemanticTokens { get; set; } The most semantic tokens generated (750: 30 seconds at 25 Hz, §5).
int NumCodebooks { get; set; } Q, the RVQ levels modeled (12).
int SampleRate { get; set; } The codec sample rate; must match SoundStream.SampleRate.
int SamplingSeed { get; set; } Seed of synthesis’s and training’s random draws.
int SemanticCodes { get; set; } Semantic token vocabulary (1024 w2v-BERT k-means clusters).
int SemanticFrameRate { get; set; } Semantic tokens per second (25); each is duplicated to the codec’s frame rate.
int TextFeedForwardDim { get; set; } ByT5’s feed-forward width (3840).
int TextKeyValueDim { get; set; } ByT5’s attention head width (64).
int TextModelDim { get; set; } ByT5’s width (1536).
int TopK { get; set; } Semantic tokens are sampled from the top K (125).
int TrainingSteps { get; set; } Total training steps the cosine decay spans.
int VocabSize { get; set; } The derived ByT5 byte and semantic token vocabulary.
int WarmupSteps { get; set; } Linear warmup steps.

Methods

Method Summary
SoundStormOptions DialogueSynthesis(int[] codecStrides) The paper’s dialogue system (§5): semantic tokens from 4096 k-means clusters of a 0.6B BEST-RQ at 25 Hz, over a 24 kHz SoundStream at 50 frames per second with 12 RVQ levels.