W2vBertOptions
Options & Configuration · AiDotNet.Audio.Foundations
Options for w2v-BERT (Chung et al. 2021, “w2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training”).
Inheritance: ModelOptions → W2vBertOptions
For Beginners
w2v-BERT learns what speech sounds like from recordings alone, with no transcripts. Its middle layers make good “what is being said” tokens for speech generators such as AudioLM and SoundStorm.
How It Works
The defaults are w2v-BERT XL (Table 1, 0.6B parameters): 80-bin log-mel features (25 ms windows every 10 ms, as the Conformer setup it follows uses), a feature encoder of two stride-2 2-D convolutions (4× fewer frames, so 25 frames a second), a contrastive module of a linear projection and 12 Conformer blocks, and a masked-prediction module of 12 more, all 1024 wide with 8 heads, convolution kernel 5 and no relative attention. The quantizer has one codebook of 1024 entries 1024 wide (Table 1) and otherwise follows wav2vec 2.0’s (§3.2: K = 100 distractors, diversity weight α = 0.1, κ = 0.1, the Gumbel temperature annealed from 2 to 0.5). Masking starts a 10-step span at each step with probability 0.065, and a masked step is replaced by a random vector (§4.2). The loss is L_c + L_m (β = γ = 1). Pre-training uses Adam with the Transformer schedule (Vaswani et al. 2017, §5.3: β = 0.9 / 0.98, ε = 1e-9) at a peak of 2e-3 after 25k warm-up steps (§4.2).
Not stated by the paper or the works it follows, and decided here: the convolutions’ 3×3 kernels and their channel
count (the model width), as Conformer’s convolutional subsampling is reproduced by ESPnet’s
Conv2dSubsampling; the random vectors are drawn from N(0, 1); the masked-prediction loss is summed over the
masked steps, as wav2vec 2.0’s contrastive loss is; and the log-mel floor (LogMelFloor).
Google released no w2v-BERT code or weights. Meta’s w2v-BERT 2.0 (facebook/w2v-bert-2.0) is a different
network (stacked-frame features, relative-key attention, one 24-layer stack), so its checkpoint does not load here.
Constructors
| Constructor | Summary |
|---|---|
W2vBertOptions() |
Creates w2v-BERT XL’s options. |
W2vBertOptions(W2vBertOptions other) |
Creates a copy of other. |
Properties
| Property | Summary |
|---|---|
OnnxModelOptions OnnxOptions { get; set; } |
ONNX runtime options. |
double DiversityWeight { get; set; } |
Codebook diversity loss weight α (0.1). |
double LearningRate { get; set; } |
Peak learning rate (2e-3). |
double LogMelFloor { get; set; } |
The floor added before the log, log(mel + floor). |
double MaskProbability { get; set; } |
Probability that a step starts a masked span (0.065). |
int CodeDim { get; set; } |
Quantized vector width (1024). |
int CodebookSize { get; set; } |
Quantizer codebook entries, which are the masked-prediction targets (1024). |
int ContrastiveLayers { get; set; } |
Conformer blocks in the contrastive module (12). |
int ConvKernel { get; set; } |
Conformer convolution kernel (5). |
int FeedForwardMultiplier { get; set; } |
Feed-forward width as a multiple of the model width (4, Conformer’s). |
int HopLength { get; set; } |
Frame step in samples (160: 10 ms). |
int MaskLength { get; set; } |
Steps in a masked span (10). |
int MaskedLayers { get; set; } |
Conformer blocks in the masked-prediction module (12 for XL, 30 for XXL). |
int MelBins { get; set; } |
Log-mel filterbank bins (80). |
int ModelDim { get; set; } |
Model width (1024). |
int Negatives { get; set; } |
Distractors per masked step (100, wav2vec 2.0’s). |
int NumHeads { get; set; } |
Attention heads (8). |
int SampleRate { get; set; } |
Audio sample rate (16 kHz). |
int SemanticClusters { get; set; } |
k-means clusters of the semantic tokens (1024, AudioLM’s). |
int SemanticLayer { get; set; } |
The masked-prediction block, counted from 1, whose output AudioLM clusters into semantic tokens (7). |
int WarmupSteps { get; set; } |
Transformer-schedule warm-up steps (25 000). |
int WindowLength { get; set; } |
Analysis window in samples (400: 25 ms). |
int? FixedDraw { get; set; } |
A seed that fixes the pre-training draw (mask, Gumbel noise, distractors, dropout) for every step and every evaluated objective, and keeps each input’s codebook entries from its first evaluation; null, the paper’s setting, draws afresh each… |
string ModelPath { get; set; } |
Path of an ONNX export, for inference. |
Methods
| Method | Summary |
|---|---|
W2vBertOptions Xxl() |
w2v-BERT XXL (Table 1, 1.0B parameters): 30 masked-prediction blocks. |