SoundStream<T>
Models & Types · AiDotNet.Audio.Generation
SoundStream: a fully convolutional, causal encoder–decoder whose embeddings are compressed by a residual vector quantizer trained with quantizer dropout, so one model serves every bitrate up to its maximum, trained adversarially against a wave-based and an STFT-based discriminator, with optional denoising through FiLM conditioning.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → NeuralAudioCodecBase<T> → SoundStream<T>
For Beginners
SoundStream squeezes audio into a short stream of codes and rebuilds it. Train it once and choose the bitrate later by using more or fewer codebooks; it can also clean up background noise as it compresses.
How It Works
Reference: “SoundStream: An End-to-End Neural Audio Codec” (Zeghidour et al., 2021). No official implementation exists; where the paper is silent this follows EnCodec’s reimplementation of SoundStream (Défossez et al. 2022, App. A.2), which used EnCodec’s training setup.
The encoder (§III-B) is a causal k = 7 convolution to C = 32 channels, four blocks of three residual units (dilations 1, 3, 9; kernels 7 then 1) and a strided convolution (kernel 2S; strides 2, 4, 5, 8) doubling the channels, and a k = 3 convolution to D; the decoder mirrors it (k = 7 to 16C, transposed convolutions, k = 7 to the waveform). ELU, no normalization. The residual vector quantizer (§III-C) has N_q codebooks of 1024 entries with EMA updates, k-means initialization and dead-code replacement; each training example uses n_q ~ U[1, N_q] of them. FiLM layers (§III-F) scale and shift the bottleneck from a one-hot denoising flag.
Training (§III-E) minimizes λ_adv L_adv + λ_feat L_feat + λ_rec L_rec (1, 100, 1; Eq. 6): the hinge adversarial loss averaged over the STFT discriminator and three MelGAN wave discriminators (Eq. 1–2), the mean L1 distance between their internal features (Eq. 3), and the multi-scale mel loss over windows 2⁶ … 2¹¹ (Eq. 4–5).
Constructors
| Constructor | Summary |
|---|---|
SoundStream(NeuralNetworkArchitecture<T> architecture, SoundStreamOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a trainable SoundStream. |
SoundStream(NeuralNetworkArchitecture<T> architecture, string modelPath, SoundStreamOptions options) |
Creates a trainable SoundStream. |
Properties
| Property | Summary |
|---|---|
bool Denoise { get; set; } |
Gets or sets whether coding applies the denoising conditioning (§III-F). |
int HopLength { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
void TrainDenoising(Tensor<T> input, Tensor<T> target, bool denoise) |
One training step on a (inputs, targets, denoise) tuple (§III-F): with denoise false the target is the input itself; with it true, the clean component of a noisy input. |