NeMoMultitask<T>
Models & Types · AiDotNet.SpeechRecognition.NeMo
NeMo’s multitask attention encoder-decoder, Canary-1B (Puvvada et al. 2024): a Fast Conformer encoder and a Transformer decoder.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → NeMoMultitask<T>
Implements: ISpeechRecognizer<T>
For Beginners
NeMo multitask AED (Canary-1B) turns 16 kHz audio into an 80-band log-mel spectrogram, shortens it 8x with depthwise-separable convolutions, and encodes it with Conformer blocks, each mixing self-attention (long-range context) with a convolution (local patterns). A Transformer decoder writes the transcript token by token while reading the encoded audio.
How It Works
The defaults are Canary-1B (section 3): a Fast Conformer XL encoder (8x subsampling, 24 blocks of width 1024, 4096 feed-forward units, kernel 9, 8 heads) over 128-dimensional log-mel features, and a 24-layer Transformer decoder (width 1024, 4096 feed-forward units, 8 heads, fixed positional encoding) trained with cross-entropy. The vocabulary is 4,128 tokens: four 1024-token language tokenizers and 32 special tokens. The decoder here reads only its start token, not the language, task and punctuation prompt. SpecAugment masks the features in training (section 3.1). Training minimizes the decoder’s cross-entropy and transcription decodes greedily. The paper’s blocks use relative
positional encoding; these use rotary position encoding, the relative scheme
ConformerBlockLayer provides.
Example
var architecture = new NeuralNetworkArchitecture<float>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.SpeechRecognition,
inputSize: 16000, outputSize: 34);
var model = new NeMoMultitask<float>(architecture);
var text = model.Transcribe(waveform).Text;
Constructors
| Constructor | Summary |
|---|---|
NeMoMultitask(NeuralNetworkArchitecture<T> architecture, NeMoMultitaskOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
NeMoMultitask(NeuralNetworkArchitecture<T> architecture, string modelPath, NeMoMultitaskOptions options) |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
|
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
Transcribes audio: the encoder and the decoder with greedy decoding. |
string DetectLanguage(Tensor<T> audio) |