EmformerRNNT<T>

Models & Types · AiDotNet.SpeechRecognition.Streaming

Emformer RNN-T (Shi et al. 2021): a streaming Transformer transducer whose attention for each segment covers a bounded left context, the segment and a short look-ahead.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → EmformerRNNT<T>

Implements: ISpeechRecognizer<T>

For Beginners

Emformer RNN-T listens to audio in short segments, so it can transcribe while you speak: each segment sees a little of the past and a few frames of the future, and a small recurrent network turns the encoded audio into text token by token.

How It Works

The network is torchaudio’s emformer_rnnt_base: 80-dimensional log-mel frames through a bias-free linear layer to 128 dimensions, time reduction by stacking 4 frames (width 512), 20 Emformer layers with 8 heads, 2048 GELU feed-forward units, segments of 16 frames (4 after reduction), a 4-frame look-ahead (1 after reduction) and 30 frames of left context, an output linear layer to 1024 with layer norm, a prediction network of a 512-dimensional symbol embedding and three layer-normalized LSTMs of 512 units between layer norms, and a joiner applying ReLU to the summed encodings. Each layer processes the look-ahead frames as separate right-context rows, as the paper’s training does; the memory bank is off, as in the base configuration. Training uses the transducer loss over SpecAugmented features.

Example

var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.SpeechRecognition,
    inputSize: 16000, outputSize: 4097);
var model = new EmformerRNNT<float>(architecture);
var text = model.Transcribe(waveform).Text;

Constructors

Constructor Summary
EmformerRNNT(NeuralNetworkArchitecture<T> architecture, EmformerRNNTOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
EmformerRNNT(NeuralNetworkArchitecture<T> architecture, string modelPath, EmformerRNNTOptions options)

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes audio: the encoder and the transducer with greedy decoding.
string DetectLanguage(Tensor<T> audio)