ParakeetCTC<T>

Models & Types · AiDotNet.SpeechRecognition.NeMo

Parakeet-CTC: NVIDIA NeMo’s Fast Conformer (Rekesh et al. 2023) with a CTC layer.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → ParakeetCTC<T>

Implements: ISpeechRecognizer<T>

For Beginners

Parakeet-CTC turns 16 kHz audio into an 80-band log-mel spectrogram, shortens it 8x with depthwise-separable convolutions, and encodes it with Conformer blocks, each mixing self-attention (long-range context) with a convolution (local patterns). A linear layer gives every frame a probability for each token; collapsing repeats and blanks gives the transcript.

How It Works

Fast Conformer replaces the Conformer front end with 8x depthwise-separable convolution subsampling of 256 channels and narrows the depthwise kernel to 9 (section 3.1). The defaults are parakeet-ctc-0.6b: 24 blocks, width 1024, 8 heads, a 1024-token vocabulary and a CTC layer. SpecAugment masks the features in training (section 3.1). The CTC layer trains with the CTC loss (Graves et al. 2006) and decodes greedily. The paper’s blocks use relative positional encoding; these use rotary position encoding, the relative scheme ConformerBlockLayer provides.

Example

var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.SpeechRecognition,
    inputSize: 16000, outputSize: 34);
var model = new ParakeetCTC<float>(architecture);
var text = model.Transcribe(waveform).Text;

Constructors

Constructor Summary
ParakeetCTC(NeuralNetworkArchitecture<T> architecture, ParakeetCTCOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
ParakeetCTC(NeuralNetworkArchitecture<T> architecture, string modelPath, ParakeetCTCOptions options)

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes audio: the encoder and the CTC layer with greedy decoding.
string DetectLanguage(Tensor<T> audio)