WavLM<T>

Models & Types · AiDotNet.Audio.Foundations

WavLM self-supervised speech representation model from Microsoft.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → WavLM<T>

Implements: IAudioFoundationModel<T>, IPaperOptimizerVariant

For Beginners

WavLM improves on HuBERT by also learning to handle noisy audio. During training, it was given audio with added noise and had to understand it anyway. This makes it especially good at tasks in real-world noisy conditions, like telling speakers apart in a meeting or understanding speech in a crowded room.

Usage:

How It Works

WavLM (Chen et al., 2022, Microsoft) extends HuBERT with gated relative position bias and denoising pre-training objectives. It achieves state-of-the-art on the SUPERB benchmark and is particularly strong for speaker-related tasks due to its denoising training.

Constructors

Constructor Summary
WavLM(NeuralNetworkArchitecture<T> architecture, WavLMOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
WavLM(NeuralNetworkArchitecture<T> architecture, string modelPath, WavLMOptions options)

Properties

Property Summary
int EmbeddingDimension { get; }
int NumLayers { get; }
string PaperOptimizerVariant { get; } The variant this instance was built at, keying the paper recipe for it.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Task<Tensor<T>> ExtractEmbeddingsAsync(Tensor<T> audio, CancellationToken cancellationToken)
Tensor<T> ExtractEmbeddings(Tensor<T> audio)
Tensor<T> ExtractLayerFeatures(Tensor<T> audio, int layerIndex)
Tensor<T> ExtractWeightedFeatures(Tensor<T> audio, T[] layerWeights)
Tensor<T> ForwardForTraining(Tensor<T> input)
void LoadTorchWeights(Func<string, int[], double[]> read, Func<string, bool> has) Loads a transformers WavLMModel state dict straight from source.