Wav2Vec2<T>

Models & Types · AiDotNet.Audio.Foundations

wav2vec 2.0 self-supervised speech representation model.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → Wav2Vec2<T>

Implements: IAudioFoundationModel<T>, IPaperOptimizerVariant

For Beginners

wav2vec 2.0 was the breakthrough that showed AI could learn to understand speech with very little labeled data. It works by:

  1. Converting raw audio into features with a CNN
  2. Masking some features (hiding them)
  3. Learning to predict the masked parts from context

This is similar to how GPT predicts the next word, but for audio.

Usage:

How It Works

wav2vec 2.0 (Baevski et al., 2020, Meta) learns speech representations via contrastive learning over quantized latent speech units. With just 10 minutes of labeled data, it achieves WER 1.8% on LibriSpeech test-clean when fine-tuned for ASR. It pioneered the self-supervised approach later extended by HuBERT and WavLM.

Constructors

Constructor Summary
Wav2Vec2(NeuralNetworkArchitecture<T> architecture, Wav2Vec2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
Wav2Vec2(NeuralNetworkArchitecture<T> architecture, string modelPath, Wav2Vec2Options options)

Properties

Property Summary
int EmbeddingDimension { get; }
int NumLayers { get; }
string PaperOptimizerVariant { get; } The variant this instance was built at, keying the paper recipe for it.

Methods

Method Summary
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
ModelMetadata<T> GetModelMetadata()
Task<Tensor<T>> ExtractEmbeddingsAsync(Tensor<T> audio, CancellationToken cancellationToken)
Tensor<T> ExtractEmbeddings(Tensor<T> audio)
Tensor<T> ExtractLayerFeatures(Tensor<T> audio, int layerIndex)
Tensor<T> ExtractWeightedFeatures(Tensor<T> audio, T[] layerWeights)
Tensor<T> ForwardForTraining(Tensor<T> input)
void LoadTorchWeights(Func<string, int[], double[]> read, bool pretrainingCheckpoint) Loads a transformers Wav2Vec2ForPreTraining (or Wav2Vec2Model) state dict straight from source.