WavLMRobust<T>

Models & Types · AiDotNet.SpeechRecognition.Robust

WavLM-Robust: noise-robust speech recognition via denoising pre-training

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → WavLMRobust<T>

Implements: ISpeechRecognizer<T>

For Beginners

WavLM-Robust is a specialized ASR model leveraging WavLM’s unique denoising pre-training for robust speech recognition. During pre-training, overlapping utterances from different speakers are used as input, and the model learns to denoise and sepa…

How It Works

References:

  • Paper: “WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing” (Chen et al., Microsoft, 2022)

WavLM-Robust is a specialized ASR model leveraging WavLM’s unique denoising pre-training for robust speech recognition. During pre-training, overlapping utterances from different speakers are used as input, and the model learns to denoise and separate while predicting masked tokens. This produces representations inherently robust to noise, reverberation, and overlapping speech. Fine-tuned with CTC for ASR, WavLM-Robust achieves strong performance on noisy benchmarks like CHiME-4 and VoiceBank-DEMAND.

Example

// Create a WavLM-Robust model for noise-robust speech recognition
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.SpeechRecognition,
    inputHeight: 16000, inputWidth: 1, inputDepth: 1, outputSize: 5000);
var model = new WavLMRobust<double>(architecture);
            
// Or load a pre-trained ONNX model for denoising-pretrained ASR
var onnxModel = new WavLMRobust<double>(architecture, "wavlmrobust.onnx");

Constructors

Constructor Summary
WavLMRobust(NeuralNetworkArchitecture<T> architecture, WavLMRobustOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
WavLMRobust(NeuralNetworkArchitecture<T> architecture, string modelPath, WavLMRobustOptions options)

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes audio using WavLM’s denoising-pretrained encoder with CTC.
string DetectLanguage(Tensor<T> audio)
void LoadTorchWeights(Func<string, int[], double[]> read, Func<string, bool> has) Loads a transformers WavLMForCTC state dict straight from source.