NoiseRobustASR<T>
Models & Types · AiDotNet.SpeechRecognition.Robust
Noise-robust speech recognition as Whisper (Radford et al. 2022) achieves it: large-scale weakly supervised training, here at large-v3.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → NoiseRobustASR<T>
Implements: ISpeechRecognizer<T>
For Beginners
Noise-robust ASR listens to 16 kHz audio in 30-second windows turned into a log-mel spectrogram, encodes each window with a Transformer, and a second Transformer writes the transcript token by token while reading the encoded audio.
How It Works
Radford et al. attribute the robustness of Whisper to noise and domain shift to training on 680,000 hours of diverse weakly supervised audio rather than to a special architecture (sections 3.5 and 3.7). The network is the openai-whisper large-v3 (32 encoder and 32 decoder layers, width 1280, 20 heads, 128 mel bins, 51,866 tokens). Every forward reads the 30-second window openai-whisper builds (pad_or_trim); Predict returns the
decoder’s logits over the start sequence (start of transcript, language, transcribe, no timestamps), whose last row is
the first token’s distribution, and Train minimizes the teacher-forced cross-entropy of a transcript given as
token ids or as one distribution per token.
Example
var architecture = new NeuralNetworkArchitecture<float>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.SpeechRecognition,
inputSize: 480000, outputSize: 51866);
var model = new NoiseRobustASR<float>(architecture);
var text = model.Transcribe(waveform).Text;
Constructors
| Constructor | Summary |
|---|---|
NoiseRobustASR(NeuralNetworkArchitecture<T> architecture, NoiseRobustASROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
NoiseRobustASR(NeuralNetworkArchitecture<T> architecture, string modelPath, NoiseRobustASROptions options) |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
|
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
Transcribes 16 kHz audio with openai-whisper’s transcribe: 30-second windows, temperature fallback, and word timing when timestamps are requested. |
string DetectLanguage(Tensor<T> audio) |