WavLM<T>
Models & Types · AiDotNet.Audio.Foundations
WavLM self-supervised speech representation model from Microsoft.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → WavLM<T>
Implements: IAudioFoundationModel<T>, IPaperOptimizerVariant
For Beginners
WavLM improves on HuBERT by also learning to handle noisy audio. During training, it was given audio with added noise and had to understand it anyway. This makes it especially good at tasks in real-world noisy conditions, like telling speakers apart in a meeting or understanding speech in a crowded room.
Usage:
How It Works
WavLM (Chen et al., 2022, Microsoft) extends HuBERT with gated relative position bias and denoising pre-training objectives. It achieves state-of-the-art on the SUPERB benchmark and is particularly strong for speaker-related tasks due to its denoising training.
Constructors
| Constructor | Summary |
|---|---|
WavLM(NeuralNetworkArchitecture<T> architecture, WavLMOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
WavLM(NeuralNetworkArchitecture<T> architecture, string modelPath, WavLMOptions options) |
Properties
| Property | Summary |
|---|---|
int EmbeddingDimension { get; } |
|
int NumLayers { get; } |
|
string PaperOptimizerVariant { get; } |
The variant this instance was built at, keying the paper recipe for it. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Task<Tensor<T>> ExtractEmbeddingsAsync(Tensor<T> audio, CancellationToken cancellationToken) |
|
Tensor<T> ExtractEmbeddings(Tensor<T> audio) |
|
Tensor<T> ExtractLayerFeatures(Tensor<T> audio, int layerIndex) |
|
Tensor<T> ExtractWeightedFeatures(Tensor<T> audio, T[] layerWeights) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
void LoadTorchWeights(Func<string, int[], double[]> read, Func<string, bool> has) |
Loads a transformers WavLMModel state dict straight from source. |