OmnilangualASR<T>
Models & Types · AiDotNet.SpeechRecognition.Multilingual
Omnilingual ASR’s CTC recognizer: a wav2vec 2.0 encoder over the raw waveform with a single linear CTC layer.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → OmnilangualASR<T>
Implements: ISpeechRecognizer<T>
For Beginners
The model listens to raw 16 kHz audio and outputs, for every 20 ms frame, a probability for each character (plus a “blank”). Collapsing repeats and dropping blanks gives the transcript.
How It Works
References:
- Paper: “Omnilingual ASR: Open-Source Multilingual Speech Recognition for 1600+ Languages”
(Omnilingual ASR team, Meta, 2025)
Omnilingual ASR pre-trains wav2vec 2.0 encoders of 0.3B, 1B, 3B and 7B parameters on 4.3M hours of unlabeled speech in over 1,600 languages (section 4.1), then builds two recognizers on them (section 4.2): a CTC model that adds a single linear layer and fine-tunes the whole network with a CTC loss, and an LLM-ASR model with a Transformer decoder. This class is the CTC model. Its defaults are the 300M encoder: 24 layers, 1024 wide, 16 heads, 4096 feed-forward (Table 4).
Example
// Create an Omnilingual ASR model for universal multilingual recognition
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.SpeechRecognition,
inputHeight: 16000, inputWidth: 1, inputDepth: 1, outputSize: 9812);
var model = new OmnilangualASR<double>(architecture);
// Or load an exported ONNX model
var onnxModel = new OmnilangualASR<double>(architecture, "omnilangual.onnx");
Constructors
| Constructor | Summary |
|---|---|
OmnilangualASR(NeuralNetworkArchitecture<T> architecture, OmnilangualASROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
OmnilangualASR(NeuralNetworkArchitecture<T> architecture, string modelPath, OmnilangualASROptions options) |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
|
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
Transcribes audio: the encoder and CTC layer give per-frame distributions, decoded greedily. |
string DetectLanguage(Tensor<T> audio) |
|
void LoadTorchWeights(Func<string, int[], double[]> read) |
Loads a transformers *ForCTC state dict straight from source. |