W2vBert<T>
Models & Types · AiDotNet.Audio.Foundations
w2v-BERT: self-supervised speech representations from contrastive learning and masked prediction together (Chung et al. 2021), and the semantic tokens AudioLM, SPEAR-TTS and SoundStorm build on.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → W2vBert<T>
Implements: IAudioFoundationModel<T>
For Beginners
w2v-BERT learns what speech sounds like from recordings alone. Its middle layers capture what is being said, which is why speech generators use them as “semantic” tokens.
How It Works
A feature encoder turns 80-bin log-mel features into 25 frames a second. Spans of them are masked (replaced by random
vectors) and passed through the contrastive module, whose context vectors must pick each masked step’s quantized
feature among distractors (wav2vec 2.0’s task); the quantizer’s choice for each step is its token id. The
masked-prediction module reads the context vectors and predicts the masked steps’ token ids. Both modules train
together on L_c + L_m (§3.2). See W2vBertOptions for the sizes and the choices the paper leaves open.
Semantic tokens (AudioLM, Borsos et al. 2022, §4): the output of the masked-prediction module’s 7th block,
standardized per dimension, clustered by k-means into 1024 tokens (FitSemanticTokenizer,
SemanticTokens).
Example
var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 1, outputSize: 1024);
var model = new W2vBert<float>(architecture, new W2vBertOptions());
foreach (var batch in unlabeledSpeech) model.Train(batch, batch); // self-supervised pre-training
model.FitSemanticTokenizer(sampleRecordings);
int[] tokens = model.SemanticTokens(recording); // 25 tokens a second
Constructors
| Constructor | Summary |
|---|---|
W2vBert(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
W2vBert(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Calls the fully specified overload with the omitted parameters’ defaults. |
W2vBert(NeuralNetworkArchitecture<T> architecture, W2vBertOptions options) |
Calls the fully specified overload with the omitted parameters’ defaults. |
W2vBert(NeuralNetworkArchitecture<T> architecture, W2vBertOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a native, trainable w2v-BERT. |
W2vBert(NeuralNetworkArchitecture<T> architecture, string modelPath) |
Calls the fully specified overload with the omitted parameters’ defaults. |
W2vBert(NeuralNetworkArchitecture<T> architecture, string modelPath, W2vBertOptions options) |
Creates a native, trainable w2v-BERT. |
Properties
| Property | Summary |
|---|---|
int EmbeddingDimension { get; } |
|
int NumLayers { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<Tensor<T>> ExtractEmbeddingsAsync(Tensor<T> audio, CancellationToken cancellationToken) |
|
Tensor<T> ExtractEmbeddings(Tensor<T> audio) |
|
Tensor<T> ExtractLayerFeatures(Tensor<T> audio, int layerIndex) |
|
Tensor<T> ExtractWeightedFeatures(Tensor<T> audio, T[] layerWeights) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
int[] SemanticTokens(Tensor<T> recording) |
The semantic tokens of a recording, one per frame at 25 Hz (FitSemanticTokenizer first). |
void FitSemanticTokenizer(IEnumerable<Tensor<T>> recordings) |
Fits AudioLM’s semantic tokenizer on recordings: the masked-prediction module’s SemanticLayer-th block output, standardized per dimension (zero mean, unit variance over the frames), clustered by k-means into SemanticClusters tokens. |