RobustConformer<T>

Models & Types · AiDotNet.SpeechRecognition.Robust

Noise-robust wav2vec 2.0 (Wang et al. 2021): contrastive pre-training continued on noisy speech with an auxiliary module that reconstructs the clean speech, then CTC fine-tuning.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → RobustConformer<T>

Implements: ISpeechRecognizer<T>

For Beginners

The model reads raw 16 kHz audio. A stack of convolutions turns it into one feature vector every 20 ms, and a Transformer gives each vector context from the whole utterance. To make those vectors survive background noise, pre-training feeds noisy audio and asks a small extra network to rebuild the clean audio from them; that extra network is dropped afterwards. For recognition a linear layer reads each vector as character probabilities, and collapsing repeats and blanks gives the transcript.

How It Works

Section 2: the wav2vec 2.0 encoder (Baevski et al. 2020), whose contrastive loss L_c and diversity loss L_d are kept, plus a reconstruction module on the contextual representations C: a two-layer BLSTM with layer normalization after each layer and a CNN decoder symmetric to the feature encoder, trained by the mean absolute error L_r to the clean waveform (Equation 3). Continual pre-training (ContinualPretrain) minimizes L = L_c + 0.1 L_d + 0.1 L_r (Equation 4) over noisy input; wav2vec 2.0’s feature penalty is kept as in its reference objective. Fine-tuning (Train) adds a linear layer for character predictions and minimizes the CTC loss (section 3.3); the reconstruction module is not used in fine-tuning or inference. The defaults are section 3.2’s BASE model. The BLSTM width is not stated; the default makes its output the encoder width. The class keeps its earlier name, from when it was a generic Conformer template.

Example

var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.SpeechRecognition,
    inputSize: 16000, outputSize: 128);
var model = new RobustConformer<float>(architecture);
model.ContinualPretrain(noisyWaveforms, cleanWaveforms);
model.Train(waveform, transcriptCodePoints);
var text = model.Transcribe(waveform).Text;

Constructors

Constructor Summary
RobustConformer(NeuralNetworkArchitecture<T> architecture, RobustConformerOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
RobustConformer(NeuralNetworkArchitecture<T> architecture, string modelPath, RobustConformerOptions options)

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes audio: the encoder and the CTC layer with greedy decoding.
string DetectLanguage(Tensor<T> audio)
void ContinualPretrain(Tensor<T> noisy, Tensor<T> clean) One continual pre-training step (section 2.3) on noisy waveforms [B, S] (or [S]) and their clean references of the same shape: L = L_c + λ1 L_d + λ2 L_r (Equation 4).