SpeechTokenizerOptions

Options & Configuration · AiDotNet.Audio.Generation

Options for SpeechTokenizer (“SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models”, Zhang et al., ICLR 2024). The defaults are the paper’s: EnCodec’s SEANet at C = 32 with a two-layer BiLSTM in the encoder, strides (2, 4, 5, 8) at 16 kHz (50 frames per second), eight 1024-entry codebooks whose first is distilled toward HuBERT representations. OfficialCheckpoint is the released speechtokenizer_hubert_avg configuration.

Inheritance: ModelOptions → NeuralAudioCodecOptions → SpeechTokenizerOptions

For Beginners

SpeechTokenizer turns speech into eight streams of codes: the first carries what is said, the other seven add who said it and how. Speech language models generate the first stream from text and the rest from the first.

Constructors

Constructor Summary
SpeechTokenizerOptions() Creates the paper’s configuration.

Properties

Property Summary
bool CommitmentAsMean { get; set; } Gets or sets whether the codebook commitment loss is the reference’s mean squared error (true: the λ below were tuned for it) rather than the sum of squared norms of §3.3.
double AdversarialLossWeight { get; set; } Gets or sets λ_g (1).
double Beta1 { get; set; } Gets or sets Adam’s β1 (0.9).
double Beta2 { get; set; } Gets or sets Adam’s β2 (0.99).
double CodebookDecay { get; set; } Gets or sets the codebooks’ EMA decay (0.99).
double CommitmentLossWeight { get; set; } Gets or sets λ_w (10).
double DistillationLossWeight { get; set; } Gets or sets λ_distill (120).
double FeatureLossWeight { get; set; } Gets or sets λ_feat (1).
double FrequencyLossWeight { get; set; } Gets or sets λ_f on the multi-scale mel loss (45).
double LearningRate { get; set; } Gets or sets Adam’s learning rate (4e-4: the paper’s maximum).
double LearningRateDecay { get; set; } Gets or sets the learning rate’s decay per epoch (0.98).
double TimeLossWeight { get; set; } Gets or sets λ_t on the mean absolute time-domain error (500).
int DeadCodeThreshold { get; set; } Gets or sets the EMA usage below which a code is replaced by a batch vector (2).
int Dimension { get; set; } Gets or sets the latent dimension D (the paper leaves it unstated; 1024, the reference’s).
int DiscriminatorWidthDivisor { get; set; } Gets or sets a divisor on the period and scale discriminators’ widths (1).
int Filters { get; set; } Gets or sets the base channel count C (32; App.
int KMeansIterations { get; set; } Gets or sets the k-means iterations of the first-batch codebook initialization (50).
int LstmLayers { get; set; } Gets or sets the encoder’s BiLSTM layers (2).
int MelBins { get; set; } Gets or sets the mel bins (64).
int SemanticDimension { get; set; } Gets or sets the semantic teacher’s feature dimension (768: HuBERT base).
int StftDiscriminatorFilters { get; set; } Gets or sets the MS-STFT discriminator’s channels (32).
int UpdatesPerEpoch { get; set; } Gets or sets the optimizer steps per epoch over which the decay is applied (1000).
int[] DiscriminatorPeriods { get; set; } Gets or sets the multi-period discriminator’s periods (2, 3, 5, 7, 11).
int[] MelScales { get; set; } Gets or sets the mel losses’ scales i (window 2^i, hop 2^i / 4): 5 … 11.
int[] Ratios { get; set; } Gets or sets the strides in decoder order (8, 5, 4, 2); the encoder applies them reversed (2, 4, 5, 8).
int[] StftDiscriminatorWindows { get; set; } Gets or sets the MS-STFT discriminator’s windows (1024, 2048, 512, 256, 128; hop a quarter; reference).

Methods

Method Summary
SpeechTokenizerOptions OfficialCheckpoint() The released speechtokenizer_hubert_avg configuration: C = 64.