BESTRQ<T>
Models & Types · AiDotNet.SpeechRecognition.Foundation
BEST-RQ: a Conformer speech encoder pre-trained to predict the labels a fixed random-projection quantizer assigns to the masked input, with a CTC layer for recognition.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → BESTRQ<T>
Implements: ISpeechRecognizer<T>
For Beginners
BEST-RQ learns from unlabeled speech by hiding parts of it and guessing what was hidden. The “answers” come from a random matrix and a random codebook that are never trained: each chunk of the original audio features is projected by the matrix and labelled with its nearest codebook entry. After this pre-training the encoder is fine-tuned on transcribed speech.
How It Works
References:
- Paper: “Self-supervised Learning with Random-Projection Quantizer for Speech Recognition” (Chiu et al., Google, 2022)
The network is the paper’s 0.6B non-streaming model (section 4.1.1): normalized 80-bin log-mel input at a 10 ms stride,
two stride-2 convolutions for 4x reduction, and 24 Conformer XL blocks (Zhang et al. 2020: 1024 wide, 8 heads, kernel 5,
no relative positional embedding). Pretrain runs the masked-prediction objective of section 3.2; ordinary
training fine-tunes through a projection layer and a CTC layer (the paper fine-tunes an RNN-T decoder instead).
Example
// Create a BEST-RQ recognizer (Conformer XL with a CTC layer)
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.SpeechRecognition,
inputHeight: 16000, inputWidth: 1, inputDepth: 1, outputSize: 1024);
var model = new BESTRQ<double>(architecture);
// Or load an exported ONNX model
var onnxModel = new BESTRQ<double>(architecture, "bestrq.onnx");
Constructors
| Constructor | Summary |
|---|---|
BESTRQ(NeuralNetworkArchitecture<T> architecture, BESTRQOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
BESTRQ(NeuralNetworkArchitecture<T> architecture, string modelPath, BESTRQOptions options) |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
|
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
ModelMetadata<T> GetModelMetadata() |
|
T Pretrain(Tensor<T> audio) |
One self-supervised pre-training step (section 3.2) on waveforms [S] or [batch, S]. |
T PretrainingLoss(Tensor<T> audio, Random random) |
The masked-prediction pre-training loss of waveforms [S] or [batch, S] (Chiu et al. |
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
Transcribes audio: the encoder, projection and CTC layer give per-frame distributions, decoded greedily. |
string DetectLanguage(Tensor<T> audio) |