CodeSwitchingASR<T>
Models & Types · AiDotNet.SpeechRecognition.Specialized
Code-switching ASR: a hybrid CTC/attention model trained jointly with an EXPLICIT language-identification task, for speech that mixes languages inside one utterance.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → CodeSwitchingASR<T>
Implements: ISpeechRecognizer<T>
For Beginners
Bilingual speakers often switch language mid-sentence. This model listens with a bidirectional recurrent encoder, writes the transcript one token at a time while looking back at the audio, and in parallel says which language each token is in. Being forced to notice the switch makes the transcription better.
How It Works
Zeng, Khassanov, Pham, Xu, Chng and Li, “On the End-to-End Solution to Mandarin-English Code-switching Speech Recognition” (Interspeech 2019, arXiv:1811.00241). Code-switching “refers to a linguistic phenomenon where a speaker uses different languages in an utterance or between alternating utterances”. The paper’s headline result is that making the model say WHICH language it is hearing, as a separate supervised task, improves the transcription itself:
The network is section 4.3’s ESPnet model: log-mel features, a CNN front end (ESPnet’s VGG2L, 4x in time), a six-layer BLSTM encoder of 320 units, a one-layer 320-unit LSTM decoder with location-aware attention (content plus location, Chorowski et al. 2015), and a CTC layer on the encoder. The LID task is a second decoder that emits the language Z of each output token from the same attention (Figure 1, LID_shared) or its own (LID_indep), so its labels come from the transcript: Mandarin characters and English BPE units occupy disjoint token ranges (“Mandarin uses characters while English uses BPE units”). Attention sizes not stated by the paper are ESPnet’s defaults. Transcription decodes greedily with the attention decoder; the language reported is the LID decoder’s.
Example
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.SpeechRecognition,
inputSize: 16000, outputSize: 6002);
var model = new CodeSwitchingASR<double>(architecture);
model.Train(waveform, transcriptTokenIds);
var result = model.Transcribe(waveform);
var language = model.DetectLanguage(waveform); // from the learned LID decoder
Constructors
| Constructor | Summary |
|---|---|
CodeSwitchingASR(NeuralNetworkArchitecture<T> architecture, CodeSwitchingASROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
CodeSwitchingASR(NeuralNetworkArchitecture<T> architecture, string modelPath, CodeSwitchingASROptions options) |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
|
int EnglishTokenOffset { get; } |
Gets the token id at which the English BPE inventory begins. |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
The LID decoder’s language posteriors averaged over the greedy decoding’s steps (the step that ends the transcript included) and normalized over the two languages. |
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
Transcribes audio with greedy attention decoding; the language is the LID decoder’s. |
string DetectLanguage(Tensor<T> audio) |
Reports the dominant language from the LEARNED LID decoder. |
string LanguageOfToken(int tokenId) |
Returns the language a token id belongs to, which is well-defined precisely because the two inventories are concatenated rather than unified. |