NeMoMultitask<T>

Models & Types · AiDotNet.SpeechRecognition.NeMo

NeMo’s multitask attention encoder-decoder, Canary-1B (Puvvada et al. 2024): a Fast Conformer encoder and a Transformer decoder.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → NeMoMultitask<T>

Implements: ISpeechRecognizer<T>

For Beginners

NeMo multitask AED (Canary-1B) turns 16 kHz audio into an 80-band log-mel spectrogram, shortens it 8x with depthwise-separable convolutions, and encodes it with Conformer blocks, each mixing self-attention (long-range context) with a convolution (local patterns). A Transformer decoder writes the transcript token by token while reading the encoded audio.

How It Works

The defaults are Canary-1B (section 3): a Fast Conformer XL encoder (8x subsampling, 24 blocks of width 1024, 4096 feed-forward units, kernel 9, 8 heads) over 128-dimensional log-mel features, and a 24-layer Transformer decoder (width 1024, 4096 feed-forward units, 8 heads, fixed positional encoding) trained with cross-entropy. The vocabulary is 4,128 tokens: four 1024-token language tokenizers and 32 special tokens. The decoder here reads only its start token, not the language, task and punctuation prompt. SpecAugment masks the features in training (section 3.1). Training minimizes the decoder’s cross-entropy and transcription decodes greedily. The paper’s blocks use relative positional encoding; these use rotary position encoding, the relative scheme ConformerBlockLayer provides.

Example

var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.SpeechRecognition,
    inputSize: 16000, outputSize: 34);
var model = new NeMoMultitask<float>(architecture);
var text = model.Transcribe(waveform).Text;

Constructors

Constructor Summary
NeMoMultitask(NeuralNetworkArchitecture<T> architecture, NeMoMultitaskOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
NeMoMultitask(NeuralNetworkArchitecture<T> architecture, string modelPath, NeMoMultitaskOptions options)

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes audio: the encoder and the decoder with greedy decoding.
string DetectLanguage(Tensor<T> audio)