CodeSwitchingASR<T>

Models & Types · AiDotNet.SpeechRecognition.Specialized

Code-switching ASR: a hybrid CTC/attention model trained jointly with an EXPLICIT language-identification task, for speech that mixes languages inside one utterance.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → CodeSwitchingASR<T>

Implements: ISpeechRecognizer<T>

For Beginners

Bilingual speakers often switch language mid-sentence. This model listens with a bidirectional recurrent encoder, writes the transcript one token at a time while looking back at the audio, and in parallel says which language each token is in. Being forced to notice the switch makes the transcription better.

How It Works

Zeng, Khassanov, Pham, Xu, Chng and Li, “On the End-to-End Solution to Mandarin-English Code-switching Speech Recognition” (Interspeech 2019, arXiv:1811.00241). Code-switching “refers to a linguistic phenomenon where a speaker uses different languages in an utterance or between alternating utterances”. The paper’s headline result is that making the model say WHICH language it is hearing, as a separate supervised task, improves the transcription itself:

The network is section 4.3’s ESPnet model: log-mel features, a CNN front end (ESPnet’s VGG2L, 4x in time), a six-layer BLSTM encoder of 320 units, a one-layer 320-unit LSTM decoder with location-aware attention (content plus location, Chorowski et al. 2015), and a CTC layer on the encoder. The LID task is a second decoder that emits the language Z of each output token from the same attention (Figure 1, LID_shared) or its own (LID_indep), so its labels come from the transcript: Mandarin characters and English BPE units occupy disjoint token ranges (“Mandarin uses characters while English uses BPE units”). Attention sizes not stated by the paper are ESPnet’s defaults. Transcription decodes greedily with the attention decoder; the language reported is the LID decoder’s.

Example

var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.SpeechRecognition,
    inputSize: 16000, outputSize: 6002);
var model = new CodeSwitchingASR<double>(architecture);
model.Train(waveform, transcriptTokenIds);
var result = model.Transcribe(waveform);
var language = model.DetectLanguage(waveform);   // from the learned LID decoder

Constructors

Constructor Summary
CodeSwitchingASR(NeuralNetworkArchitecture<T> architecture, CodeSwitchingASROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
CodeSwitchingASR(NeuralNetworkArchitecture<T> architecture, string modelPath, CodeSwitchingASROptions options)

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }
int EnglishTokenOffset { get; } Gets the token id at which the English BPE inventory begins.

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) The LID decoder’s language posteriors averaged over the greedy decoding’s steps (the step that ends the transcript included) and normalized over the two languages.
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes audio with greedy attention decoding; the language is the LID decoder’s.
string DetectLanguage(Tensor<T> audio) Reports the dominant language from the LEARNED LID decoder.
string LanguageOfToken(int tokenId) Returns the language a token id belongs to, which is well-defined precisely because the two inventories are concatenated rather than unified.