RobustConformer<T>
Models & Types · AiDotNet.SpeechRecognition.Robust
Noise-robust wav2vec 2.0 (Wang et al. 2021): contrastive pre-training continued on noisy speech with an auxiliary module that reconstructs the clean speech, then CTC fine-tuning.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → RobustConformer<T>
Implements: ISpeechRecognizer<T>
For Beginners
The model reads raw 16 kHz audio. A stack of convolutions turns it into one feature vector every 20 ms, and a Transformer gives each vector context from the whole utterance. To make those vectors survive background noise, pre-training feeds noisy audio and asks a small extra network to rebuild the clean audio from them; that extra network is dropped afterwards. For recognition a linear layer reads each vector as character probabilities, and collapsing repeats and blanks gives the transcript.
How It Works
Section 2: the wav2vec 2.0 encoder (Baevski et al. 2020), whose contrastive loss L_c and diversity loss L_d are kept,
plus a reconstruction module on the contextual representations C: a two-layer BLSTM with layer normalization after
each layer and a CNN decoder symmetric to the feature encoder, trained by the mean absolute error L_r to the clean
waveform (Equation 3). Continual pre-training (ContinualPretrain) minimizes L = L_c + 0.1 L_d + 0.1 L_r
(Equation 4) over noisy input; wav2vec 2.0’s feature penalty is kept as in its reference objective. Fine-tuning
(Train) adds a linear layer for character predictions and minimizes the CTC loss (section 3.3); the
reconstruction module is not used in fine-tuning or inference. The defaults are section 3.2’s BASE model. The BLSTM
width is not stated; the default makes its output the encoder width. The class keeps its earlier name, from when it
was a generic Conformer template.
Example
var architecture = new NeuralNetworkArchitecture<float>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.SpeechRecognition,
inputSize: 16000, outputSize: 128);
var model = new RobustConformer<float>(architecture);
model.ContinualPretrain(noisyWaveforms, cleanWaveforms);
model.Train(waveform, transcriptCodePoints);
var text = model.Transcribe(waveform).Text;
Constructors
| Constructor | Summary |
|---|---|
RobustConformer(NeuralNetworkArchitecture<T> architecture, RobustConformerOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
RobustConformer(NeuralNetworkArchitecture<T> architecture, string modelPath, RobustConformerOptions options) |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
|
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
Transcribes audio: the encoder and the CTC layer with greedy decoding. |
string DetectLanguage(Tensor<T> audio) |
|
void ContinualPretrain(Tensor<T> noisy, Tensor<T> clean) |
One continual pre-training step (section 2.3) on noisy waveforms [B, S] (or [S]) and their clean references of the same shape: L = L_c + λ1 L_d + λ2 L_r (Equation 4). |