RevAI<T>

Models & Types · AiDotNet.SpeechRecognition.ProprietaryAPI

Rev AI’s Reverb ASR (Rev 2024): the open-source engine behind Rev’s API, a WeNet joint CTC/attention Conformer with a bidirectional attention decoder.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → RevAI<T>

Implements: ISpeechRecognizer<T>

For Beginners

Reverb turns 16 kHz audio into an 80-band log-mel spectrogram, shortens it 4x with two convolutions, and encodes it with Conformer blocks, each mixing self-attention (long-range context) with a convolution (local patterns). A CTC layer reads the transcript off the frames, and two small decoders, one reading left to right and one right to left, learn to write it too, which makes the encoder better and lets the decoders rescore its guesses.

How It Works

Section 2.2 of the Reverb report: an 18-layer Conformer encoder and a bidirectional attention decoder of 6 Transformer layers, 3 in each direction, about 280M parameters, trained with WeNet’s joint CTC/attention objective (U2++). The defaults use width 640 (8 heads, feed-forward 2,560), the width that totals about 280M with those layers, and WeNet’s defaults for the rest: a kernel-15 convolution, CTC weight 0.3, reverse weight 0.3 and label smoothing 0.1. The loss is w L_CTC + (1 - w)((1 - r) L_left-to-right + r L_right-to-left). Transcription decodes the left-to-right decoder greedily; the release also offers CTC prefix beam search with attention rescoring. The release’s language-specific layers, which control verbatimicity, are not modelled. WeNet’s blocks use relative positional encoding; these use rotary position encoding, the relative scheme ConformerBlockLayer provides.

Example

var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.SpeechRecognition,
    inputSize: 16000, outputSize: 34);
var model = new RevAI<float>(architecture);
var text = model.Transcribe(waveform).Text;

Constructors

Constructor Summary
RevAI(NeuralNetworkArchitecture<T> architecture, RevAIOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
RevAI(NeuralNetworkArchitecture<T> architecture, string modelPath, RevAIOptions options)

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes audio: the encoder and the attention decoder with greedy decoding.
string DetectLanguage(Tensor<T> audio)