Wav2Vec2<T>
Models & Types · AiDotNet.Audio.Foundations
wav2vec 2.0 self-supervised speech representation model.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → Wav2Vec2<T>
Implements: IAudioFoundationModel<T>, IPaperOptimizerVariant
For Beginners
wav2vec 2.0 was the breakthrough that showed AI could learn to understand speech with very little labeled data. It works by:
- Converting raw audio into features with a CNN
- Masking some features (hiding them)
- Learning to predict the masked parts from context
This is similar to how GPT predicts the next word, but for audio.
Usage:
How It Works
wav2vec 2.0 (Baevski et al., 2020, Meta) learns speech representations via contrastive learning over quantized latent speech units. With just 10 minutes of labeled data, it achieves WER 1.8% on LibriSpeech test-clean when fine-tuned for ASR. It pioneered the self-supervised approach later extended by HuBERT and WavLM.
Constructors
| Constructor | Summary |
|---|---|
Wav2Vec2(NeuralNetworkArchitecture<T> architecture, Wav2Vec2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
Wav2Vec2(NeuralNetworkArchitecture<T> architecture, string modelPath, Wav2Vec2Options options) |
Properties
| Property | Summary |
|---|---|
int EmbeddingDimension { get; } |
|
int NumLayers { get; } |
|
string PaperOptimizerVariant { get; } |
The variant this instance was built at, keying the paper recipe for it. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<Tensor<T>> ExtractEmbeddingsAsync(Tensor<T> audio, CancellationToken cancellationToken) |
|
Tensor<T> ExtractEmbeddings(Tensor<T> audio) |
|
Tensor<T> ExtractLayerFeatures(Tensor<T> audio, int layerIndex) |
|
Tensor<T> ExtractWeightedFeatures(Tensor<T> audio, T[] layerWeights) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
void LoadTorchWeights(Func<string, int[], double[]> read, bool pretrainingCheckpoint) |
Loads a transformers Wav2Vec2ForPreTraining (or Wav2Vec2Model) state dict straight from source. |