EmformerRNNT<T>
Models & Types · AiDotNet.SpeechRecognition.Streaming
Emformer RNN-T (Shi et al. 2021): a streaming Transformer transducer whose attention for each segment covers a bounded left context, the segment and a short look-ahead.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → EmformerRNNT<T>
Implements: ISpeechRecognizer<T>
For Beginners
Emformer RNN-T listens to audio in short segments, so it can transcribe while you speak: each segment sees a little of the past and a few frames of the future, and a small recurrent network turns the encoded audio into text token by token.
How It Works
The network is torchaudio’s emformer_rnnt_base: 80-dimensional log-mel frames through a bias-free linear layer to
128 dimensions, time reduction by stacking 4 frames (width 512), 20 Emformer layers with 8 heads, 2048 GELU feed-forward
units, segments of 16 frames (4 after reduction), a 4-frame look-ahead (1 after reduction) and 30 frames of left context,
an output linear layer to 1024 with layer norm, a prediction network of a 512-dimensional symbol embedding and three
layer-normalized LSTMs of 512 units between layer norms, and a joiner applying ReLU to the summed encodings. Each
layer processes the look-ahead frames as separate right-context rows, as the paper’s training does; the memory bank
is off, as in the base configuration. Training uses the transducer loss over SpecAugmented features.
Example
var architecture = new NeuralNetworkArchitecture<float>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.SpeechRecognition,
inputSize: 16000, outputSize: 4097);
var model = new EmformerRNNT<float>(architecture);
var text = model.Transcribe(waveform).Text;
Constructors
| Constructor | Summary |
|---|---|
EmformerRNNT(NeuralNetworkArchitecture<T> architecture, EmformerRNNTOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
EmformerRNNT(NeuralNetworkArchitecture<T> architecture, string modelPath, EmformerRNNTOptions options) |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
|
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
Transcribes audio: the encoder and the transducer with greedy decoding. |
string DetectLanguage(Tensor<T> audio) |