NoiseRobustASR<T>

Models & Types · AiDotNet.SpeechRecognition.Robust

Noise-robust speech recognition as Whisper (Radford et al. 2022) achieves it: large-scale weakly supervised training, here at large-v3.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → NoiseRobustASR<T>

Implements: ISpeechRecognizer<T>

For Beginners

Noise-robust ASR listens to 16 kHz audio in 30-second windows turned into a log-mel spectrogram, encodes each window with a Transformer, and a second Transformer writes the transcript token by token while reading the encoded audio.

How It Works

Radford et al. attribute the robustness of Whisper to noise and domain shift to training on 680,000 hours of diverse weakly supervised audio rather than to a special architecture (sections 3.5 and 3.7). The network is the openai-whisper large-v3 (32 encoder and 32 decoder layers, width 1280, 20 heads, 128 mel bins, 51,866 tokens). Every forward reads the 30-second window openai-whisper builds (pad_or_trim); Predict returns the decoder’s logits over the start sequence (start of transcript, language, transcribe, no timestamps), whose last row is the first token’s distribution, and Train minimizes the teacher-forced cross-entropy of a transcript given as token ids or as one distribution per token.

Example

var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.SpeechRecognition,
    inputSize: 480000, outputSize: 51866);
var model = new NoiseRobustASR<float>(architecture);
var text = model.Transcribe(waveform).Text;

Constructors

Constructor Summary
NoiseRobustASR(NeuralNetworkArchitecture<T> architecture, NoiseRobustASROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
NoiseRobustASR(NeuralNetworkArchitecture<T> architecture, string modelPath, NoiseRobustASROptions options)

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes 16 kHz audio with openai-whisper’s transcribe: 30-second windows, temperature fallback, and word timing when timestamps are requested.
string DetectLanguage(Tensor<T> audio)