SoundStorm<T>
Models & Types · AiDotNet.TextToSpeech.CodecBased
SoundStorm: efficient parallel audio generation (Borsos et al. 2023), a bidirectional Conformer that fills in a SoundStream codec’s RVQ codes from time-aligned semantic tokens, level by level and many tokens at a time.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → SoundStorm<T>
Implements: ICodecTts<T>
For Beginners
SoundStorm turns “what is said” tokens into audio-codec tokens all at once instead of one by one, which makes it about a hundred times faster than generating them in order.
How It Works
Acoustic model (§3.1–3.3, SoundStormCore): per frame, the embeddings of the Q = 12 codes
and of the semantic token are summed, so the Conformer’s sequence is as long as the number of codec frames; a dense
head per level predicts that level’s codes. A training step samples a level q, a prompt t ~ U{0, T − 1} and a
cosine-scheduled mask over the positions after t; level q is masked there, and so is every finer level, and the
loss is the cross-entropy on level q’s masked positions only. Decoding fills the levels coarse to fine: the first
with 16 confidence-based (MaskGIT) iterations, the last of them greedy, and each finer level with one greedy pass.
Text to semantic (§5): ByT5-large reads the text’s UTF-8 bytes and samples semantic tokens (temperature 0.9,
top 125), which are duplicated from 25 Hz to the codec’s 50 Hz. Only the decoder trains; the encoder is a
text-pretrained ByT5 (LoadTextEncoderWeights). The semantic tokens share the decoder’s vocabulary
after ByT5’s 384 byte entries.
Voice prompting (§4.2): a voice’s recording (ReferenceAudio) and its semantic tokens
at 25 Hz (ReferenceTokens) become the unmasked prefix of both models; without a voice the
speaker is sampled. Training needs each utterance’s codes (CodecTokens,
[frames, 1 + Q]: the semantic token at the codec’s frame rate, then the Q acoustic codes), or its semantic
tokens there and its recording, which the codec encodes.
What the paper leaves unstated, and its authors released no code for, follows lucidrains/soundstorm-pytorch: a
learned mask embedding, no dropout, convolution expansion 2, and its trainer’s optimizer (see
SoundStormOptions). The Conformer’s weights are drawn from N(0, 0.02) with zero biases, as Pheme’s
SoundStorm implementation does. MaskGIT’s confidence adds Gumbel noise annealed linearly to zero, as MaskGIT’s
reference does.
Example
var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional,
NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new SoundStorm<float>(architecture, new SoundStormOptions());
model.LoadTextEncoderWeights("byt5-large.safetensors");
Tensor<float> audio = model.Synthesize("Hello there.");
Constructors
| Constructor | Summary |
|---|---|
SoundStorm(NeuralNetworkArchitecture<T> architecture, SoundStormOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a native, trainable SoundStorm. |
SoundStorm(NeuralNetworkArchitecture<T> architecture, string modelPath, SoundStormOptions options) |
Creates a native, trainable SoundStorm. |
Properties
| Property | Summary |
|---|---|
SoundStormTrainingStage CurrentStage { get; set; } |
The network a training step updates. |
SoundStream<T> Codec { get; } |
The SoundStream codec whose codes SoundStorm models. |
int CodebookSize { get; } |
|
int CodecFrameRate { get; } |
|
int CodecTokenCodebooks { get; } |
|
int CodecTokenVocabulary { get; } |
|
int MaxTextLength { get; } |
|
int NumCodebooks { get; } |
|
int SemanticToAcousticUpdates { get; set; } |
Training steps taken by the acoustic network. |
int TextToSemanticUpdates { get; set; } |
Training steps taken by each network (they set the learning-rate schedules). |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> DecodeFromTokens(Tensor<T> tokens) |
The waveform of codec codes [frames, Q]. |
Tensor<T> EncodeToTokens(Tensor<T> audio) |
The codec’s codes [frames, Q] of a recording. |
void LoadSemanticToAcousticWeights(string path) |
Loads the acoustic model from a state dict laid out as Pheme’s TTSConformer (embedding.{l}, semantic_embedding, conformer, heads.{l}). |
void LoadTextEncoderWeights(string path) |
Loads the text-pretrained ByT5 encoder (§5) from a Hugging Face T5ForConditionalGeneration state dict (safetensors or .pt), such as google/byt5-large: its 384 byte embeddings and the encoder. |
void LoadTextToSemanticWeights(string path) |
Loads the whole text-to-semantic model, as this class saves it in Hugging Face’s layout. |