OLMoASR<T>
Models & Types · AiDotNet.SpeechRecognition.LLMIntegrated
OLMoASR (Ngo et al., Ai2 2025): Whisper’s encoder-decoder, with Whisper’s English-only tokenizer, trained from scratch on the curated open OLMoASR-Mix.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → OLMoASR<T>
Implements: ISpeechRecognizer<T>
For Beginners
OLMoASR is a fully open Whisper: the same network (a transformer encoder over a log-mel spectrogram and a transformer decoder that writes the transcript token by token) trained only on public, curated audio-transcript pairs.
How It Works
§3 of the paper: OLMoASR keeps Whisper’s architecture, tokenizer and evaluation, at the English-only scales tiny.en
through large.en (39M to 1.5B parameters); the attention adds batch padding masks to the causal mask. The network is
openai-whisper’s (OpenAiWhisperNetwork): a 30-second 80-band log-mel window, two convolutions, the
encoder and the decoder with cross-attention. The English-only models’ decoder starts from
<|startoftranscript|><|notimestamps|> (no language or task tokens) and training minimizes the
teacher-forced cross-entropy of the transcript and <|endoftext|>. Table 6: AdamW (β 0.9, 0.98, ε 1e-6,
weight decay 0.1) at 1.5e-3 for every scale, 1,049 warmup updates and linear decay, gradient norm clipped at 1.
Example
var architecture = new NeuralNetworkArchitecture<float>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.SpeechRecognition,
inputSize: 480000, outputSize: 51864);
var model = new OLMoASR<float>(architecture);
var text = model.Transcribe(waveform).Text;
Constructors
| Constructor | Summary |
|---|---|
OLMoASR(NeuralNetworkArchitecture<T> architecture, OLMoASROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
OLMoASR(NeuralNetworkArchitecture<T> architecture, string modelPath, OLMoASROptions options) |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
|
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
Transcribes the first 30 seconds of audio by greedy decoding after the prompt. |
string DetectLanguage(Tensor<T> audio) |