SoundStreamOptions

Options & Configuration · AiDotNet.Audio.Generation

Options for SoundStream (“SoundStream: An End-to-End Neural Audio Codec”, Zeghidour et al. 2021). The defaults are the paper’s bitrate-scalable 24 kHz model: C = 32, strides (2, 4, 5, 8), 24 quantizers of 1024 codes (18 kbps) trained with quantizer dropout and used at 6 kbps.

Inheritance: ModelOptions → NeuralAudioCodecOptions → SoundStreamOptions

For Beginners

SoundStream compresses audio to a few numbers per 13 ms and rebuilds it. A single trained model works at any bitrate up to 18 kbps by using more or fewer codebooks per frame, and can optionally remove background noise while it compresses.

Constructors

Constructor Summary
SoundStreamOptions() Creates the paper’s bitrate-scalable configuration.

Properties

Property Summary
SoundStreamFilmPosition FilmPosition { get; set; } Gets or sets where the FiLM denoising conditioning sits (Encoder).
bool Denoise { get; set; } Gets or sets whether inference asks for denoising (false).
bool QuantizerDropout { get; set; } Gets or sets whether each training example uses n_q ~ U[1, N_q] codebooks (quantizer dropout; true).
bool ReflectPadding { get; set; } Gets or sets whether convolutions zero-pad the past rather than reflecting the signal (false: zeros, as causal padding of the past).
double AdversarialLossWeight { get; set; } Gets or sets λ_adv (1).
double Beta1 { get; set; } Gets or sets Adam’s β1 (0.5).
double Beta2 { get; set; } Gets or sets Adam’s β2 (0.9).
double CodebookDecay { get; set; } Gets or sets the codebooks’ EMA decay (0.99).
double CommitmentLossWeight { get; set; } Gets or sets the commitment loss weight (0: Eq.
double FeatureLossWeight { get; set; } Gets or sets λ_feat (100).
double LearningRate { get; set; } Gets or sets Adam’s learning rate (3e-4; the paper states none, EnCodec’s reimplementation’s).
double ReconstructionLossWeight { get; set; } Gets or sets λ_rec (1).
int DeadCodeThreshold { get; set; } Gets or sets the EMA assignment count below which a code is replaced by a frame of the batch (2).
int DilationBase { get; set; } Gets or sets the dilation growth over a block’s residual units (3: 1, 3, 9).
int Dimension { get; set; } Gets or sets the embedding dimension D (the paper leaves its value unstated; 128, as EnCodec’s reimplementation).
int Filters { get; set; } Gets or sets the base channel count C (C_enc = C_dec = 32).
int KMeansIterations { get; set; } Gets or sets the k-means iterations of the first-batch initialization (50).
int MelBins { get; set; } Gets or sets the mel bins (64).
int ResidualLayers { get; set; } Gets or sets the residual units per block (3: dilations 1, 3, 9).
int StftChannels { get; set; } Gets or sets the STFT discriminator’s base channels C (32).
int StftHop { get; set; } Gets or sets the STFT discriminator’s hop H (256).
int StftWindow { get; set; } Gets or sets the STFT discriminator’s window W (1024).
int WaveDiscriminatorScales { get; set; } Gets or sets the wave-based discriminator’s resolutions (3: original, 2× and 4× down-sampled).
int WaveDiscriminatorWidthDivisor { get; set; } Gets or sets a divisor on the wave-based discriminators’ widths (1: the paper’s).
int[] MelScales { get; set; } Gets or sets the reconstruction loss’s scales i (window and FFT 2^i, hop 2^i / 4): 6 … 11.
int[] Ratios { get; set; } Gets or sets the strides in decoder order (8, 5, 4, 2); the encoder applies them reversed (2, 4, 5, 8).
int[] ResidualKernelSizes { get; set; } Gets or sets the residual unit’s kernels (7, then 1).
int[] WaveDiscriminatorChannels { get; set; } Gets or sets the wave-based discriminators’ grouped-convolution widths (64, 256, 1024, 1024).