SoundStorm<T>

Models & Types · AiDotNet.TextToSpeech.CodecBased

SoundStorm: efficient parallel audio generation (Borsos et al. 2023), a bidirectional Conformer that fills in a SoundStream codec’s RVQ codes from time-aligned semantic tokens, level by level and many tokens at a time.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → SoundStorm<T>

Implements: ICodecTts<T>

For Beginners

SoundStorm turns “what is said” tokens into audio-codec tokens all at once instead of one by one, which makes it about a hundred times faster than generating them in order.

How It Works

Acoustic model (§3.1–3.3, SoundStormCore): per frame, the embeddings of the Q = 12 codes and of the semantic token are summed, so the Conformer’s sequence is as long as the number of codec frames; a dense head per level predicts that level’s codes. A training step samples a level q, a prompt t ~ U{0, T − 1} and a cosine-scheduled mask over the positions after t; level q is masked there, and so is every finer level, and the loss is the cross-entropy on level q’s masked positions only. Decoding fills the levels coarse to fine: the first with 16 confidence-based (MaskGIT) iterations, the last of them greedy, and each finer level with one greedy pass.

Text to semantic (§5): ByT5-large reads the text’s UTF-8 bytes and samples semantic tokens (temperature 0.9, top 125), which are duplicated from 25 Hz to the codec’s 50 Hz. Only the decoder trains; the encoder is a text-pretrained ByT5 (LoadTextEncoderWeights). The semantic tokens share the decoder’s vocabulary after ByT5’s 384 byte entries.

Voice prompting (§4.2): a voice’s recording (ReferenceAudio) and its semantic tokens at 25 Hz (ReferenceTokens) become the unmasked prefix of both models; without a voice the speaker is sampled. Training needs each utterance’s codes (CodecTokens, [frames, 1 + Q]: the semantic token at the codec’s frame rate, then the Q acoustic codes), or its semantic tokens there and its recording, which the codec encodes.

What the paper leaves unstated, and its authors released no code for, follows lucidrains/soundstorm-pytorch: a learned mask embedding, no dropout, convolution expansion 2, and its trainer’s optimizer (see SoundStormOptions). The Conformer’s weights are drawn from N(0, 0.02) with zero biases, as Pheme’s SoundStorm implementation does. MaskGIT’s confidence adds Gumbel noise annealed linearly to zero, as MaskGIT’s reference does.

Example

var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional,
    NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new SoundStorm<float>(architecture, new SoundStormOptions());
model.LoadTextEncoderWeights("byt5-large.safetensors");
Tensor<float> audio = model.Synthesize("Hello there.");

Constructors

Constructor Summary
SoundStorm(NeuralNetworkArchitecture<T> architecture, SoundStormOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a native, trainable SoundStorm.
SoundStorm(NeuralNetworkArchitecture<T> architecture, string modelPath, SoundStormOptions options) Creates a native, trainable SoundStorm.

Properties

Property Summary
SoundStormTrainingStage CurrentStage { get; set; } The network a training step updates.
SoundStream<T> Codec { get; } The SoundStream codec whose codes SoundStorm models.
int CodebookSize { get; }
int CodecFrameRate { get; }
int CodecTokenCodebooks { get; }
int CodecTokenVocabulary { get; }
int MaxTextLength { get; }
int NumCodebooks { get; }
int SemanticToAcousticUpdates { get; set; } Training steps taken by the acoustic network.
int TextToSemanticUpdates { get; set; } Training steps taken by each network (they set the learning-rate schedules).

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> DecodeFromTokens(Tensor<T> tokens) The waveform of codec codes [frames, Q].
Tensor<T> EncodeToTokens(Tensor<T> audio) The codec’s codes [frames, Q] of a recording.
void LoadSemanticToAcousticWeights(string path) Loads the acoustic model from a state dict laid out as Pheme’s TTSConformer (embedding.{l}, semantic_embedding, conformer, heads.{l}).
void LoadTextEncoderWeights(string path) Loads the text-pretrained ByT5 encoder (§5) from a Hugging Face T5ForConditionalGeneration state dict (safetensors or .pt), such as google/byt5-large: its 384 byte embeddings and the encoder.
void LoadTextToSemanticWeights(string path) Loads the whole text-to-semantic model, as this class saves it in Hugging Face’s layout.