W2vBert<T>

Models & Types · AiDotNet.Audio.Foundations

w2v-BERT: self-supervised speech representations from contrastive learning and masked prediction together (Chung et al. 2021), and the semantic tokens AudioLM, SPEAR-TTS and SoundStorm build on.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → W2vBert<T>

Implements: IAudioFoundationModel<T>

For Beginners

w2v-BERT learns what speech sounds like from recordings alone. Its middle layers capture what is being said, which is why speech generators use them as “semantic” tokens.

How It Works

A feature encoder turns 80-bin log-mel features into 25 frames a second. Spans of them are masked (replaced by random vectors) and passed through the contrastive module, whose context vectors must pick each masked step’s quantized feature among distractors (wav2vec 2.0’s task); the quantizer’s choice for each step is its token id. The masked-prediction module reads the context vectors and predicts the masked steps’ token ids. Both modules train together on L_c + L_m (§3.2). See W2vBertOptions for the sizes and the choices the paper leaves open.

Semantic tokens (AudioLM, Borsos et al. 2022, §4): the output of the masked-prediction module’s 7th block, standardized per dimension, clustered by k-means into 1024 tokens (FitSemanticTokenizer, SemanticTokens).

Example

var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 1, outputSize: 1024);
var model = new W2vBert<float>(architecture, new W2vBertOptions());
foreach (var batch in unlabeledSpeech) model.Train(batch, batch);     // self-supervised pre-training
model.FitSemanticTokenizer(sampleRecordings);
int[] tokens = model.SemanticTokens(recording);                       // 25 tokens a second

Constructors

Constructor Summary
W2vBert(NeuralNetworkArchitecture<T> architecture) Calls the fully specified overload with the omitted parameters’ defaults.
W2vBert(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Calls the fully specified overload with the omitted parameters’ defaults.
W2vBert(NeuralNetworkArchitecture<T> architecture, W2vBertOptions options) Calls the fully specified overload with the omitted parameters’ defaults.
W2vBert(NeuralNetworkArchitecture<T> architecture, W2vBertOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a native, trainable w2v-BERT.
W2vBert(NeuralNetworkArchitecture<T> architecture, string modelPath) Calls the fully specified overload with the omitted parameters’ defaults.
W2vBert(NeuralNetworkArchitecture<T> architecture, string modelPath, W2vBertOptions options) Creates a native, trainable w2v-BERT.

Properties

Property Summary
int EmbeddingDimension { get; }
int NumLayers { get; }

Methods

Method Summary
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
ModelMetadata<T> GetModelMetadata()
Task<Tensor<T>> ExtractEmbeddingsAsync(Tensor<T> audio, CancellationToken cancellationToken)
Tensor<T> ExtractEmbeddings(Tensor<T> audio)
Tensor<T> ExtractLayerFeatures(Tensor<T> audio, int layerIndex)
Tensor<T> ExtractWeightedFeatures(Tensor<T> audio, T[] layerWeights)
Tensor<T> ForwardForTraining(Tensor<T> input)
int[] SemanticTokens(Tensor<T> recording) The semantic tokens of a recording, one per frame at 25 Hz (FitSemanticTokenizer first).
void FitSemanticTokenizer(IEnumerable<Tensor<T>> recordings) Fits AudioLM’s semantic tokenizer on recordings: the masked-prediction module’s SemanticLayer-th block output, standardized per dimension (zero mean, unit variance over the frames), clustered by k-means into SemanticClusters tokens.