RNNTransducer<T>

Models & Types · AiDotNet.Audio.SpeechRecognition

RNN-Transducer (RNN-T) streaming speech recognition model.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → RNNTransducer<T>

Implements: ISpeechRecognizer<T>

For Beginners

RNN-T is a real-time speech recognizer ideal for live transcription. Unlike batch models (like Whisper) that need the whole audio, RNN-T processes speech as it arrives - perfect for live captioning and voice assistants.

It has three parts:

  1. Encoder - Listens to the audio (converts sound to features)
  2. Predictor - Remembers what was already said (like a small language model)
  3. Joiner - Combines both to decide the next output token

Think of it like a court stenographer who listens (encoder), remembers the context (predictor), and types the next word (joiner) - all in real time.

Usage:

How It Works

RNN-Transducer (Graves, 2012; He et al., 2019) combines an audio encoder with a label prediction network and a joint network to produce a streaming ASR system. Unlike CTC, RNN-T can model output dependencies through its prediction network, achieving strong results on LibriSpeech (WER 2.0% with LM) without an external language model. It is the backbone of on-device ASR in Google’s Pixel phones and NVIDIA Riva.

Constructors

Constructor Summary
RNNTransducer(NeuralNetworkArchitecture<T> architecture, RNNTransducerOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates an RNN-T model in native training mode.
RNNTransducer(NeuralNetworkArchitecture<T> architecture, string modelPath, RNNTransducerOptions options) Creates an RNN-T model in native training mode.

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps)
TranscriptionResult<T> TranscribeWithBeamSearch(Tensor<T> audio, int beamWidth, string language, bool? includeTimestamps) Transcribes audio with the transducer beam search of Graves 2012 (Algorithm 1): hypotheses over the (frame, label) lattice merged across alignments, keeping beamWidth per frame and returning the one with the highest log-probability per la…
string DetectLanguage(Tensor<T> audio)