MedicalASR<T>

Models & Types · AiDotNet.SpeechRecognition.Specialized

Medical-conversation ASR: Chiu et al. 2018’s end-to-end recognizers for doctor-patient dialogue, offering both arms of the paper’s comparison, Listen-Attend-Spell and CTC.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → MedicalASR<T>

Implements: ISpeechRecognizer<T>

For Beginners

Two ways to turn sound into text. One (CTC) labels each slice of audio and is fast but needs clean recordings. The other (LAS) reads the whole recording first, then writes the sentence letter by letter while looking back at the audio, which copes better when people talk over each other or trail off, as they do in real consultations. This model gives you both, and defaults to the one the paper found more robust.

How It Works

Chiu, Tripathi, Chou, Co, Jaitly, Jaunzeikare, Kannan, Nguyen, Sak, Sankar, Tansuwan, Wan, Wu and Zhang, “Speech recognition for medical conversations” (Interspeech 2018, arXiv:1711.07274). The paper is a COMPARISON: “We explored both CTC and LAS systems for building speech recognition models.”

Both arms read 80 filterbank channels up to 3800 Hz from 25 ms frames every 10 ms, stacked three at a time with a stride of three (240 values every 30 ms, section 4).

Example

var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.SequenceToSequence,
    inputSize: 16000, outputSize: 128);
            
// Defaults to LAS, the paper's more noise-resilient arm.
var las = new MedicalASR<double>(architecture);
            
// The paper's bidirectional CTC model.
var ctc = new MedicalASR<double>(architecture, MedicalASROptions.ForCtc());

Constructors

Constructor Summary
MedicalASR(NeuralNetworkArchitecture<T> architecture, MedicalASROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a trainable MedicalASR model (LAS by default).
MedicalASR(NeuralNetworkArchitecture<T> architecture, string modelPath, MedicalASROptions options) Creates a trainable MedicalASR model (LAS by default).

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
MedicalAsrDecoderType DecoderType { get; } Gets which arm of the paper’s comparison this instance is built as.
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }
double CurrentSamplingProbability { get; } Gets the probability with which the next training step feeds the LAS decoder its own previous prediction: a linear ramp from 0 to ScheduledSamplingProbability, then constant.
int TimeReductionFactor { get; } Gets how many 10 ms frames one encoder step covers (the frame stacking stride).

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) The paper’s recognizers are monolingual (English medical conversations): the configured language with probability 1.
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps)
string DetectLanguage(Tensor<T> audio)