OLMoASR<T>

Models & Types · AiDotNet.SpeechRecognition.LLMIntegrated

OLMoASR (Ngo et al., Ai2 2025): Whisper’s encoder-decoder, with Whisper’s English-only tokenizer, trained from scratch on the curated open OLMoASR-Mix.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → OLMoASR<T>

Implements: ISpeechRecognizer<T>

For Beginners

OLMoASR is a fully open Whisper: the same network (a transformer encoder over a log-mel spectrogram and a transformer decoder that writes the transcript token by token) trained only on public, curated audio-transcript pairs.

How It Works

§3 of the paper: OLMoASR keeps Whisper’s architecture, tokenizer and evaluation, at the English-only scales tiny.en through large.en (39M to 1.5B parameters); the attention adds batch padding masks to the causal mask. The network is openai-whisper’s (OpenAiWhisperNetwork): a 30-second 80-band log-mel window, two convolutions, the encoder and the decoder with cross-attention. The English-only models’ decoder starts from <|startoftranscript|><|notimestamps|> (no language or task tokens) and training minimizes the teacher-forced cross-entropy of the transcript and <|endoftext|>. Table 6: AdamW (β 0.9, 0.98, ε 1e-6, weight decay 0.1) at 1.5e-3 for every scale, 1,049 warmup updates and linear decay, gradient norm clipped at 1.

Example

var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.SpeechRecognition,
    inputSize: 480000, outputSize: 51864);
var model = new OLMoASR<float>(architecture);
var text = model.Transcribe(waveform).Text;

Constructors

Constructor Summary
OLMoASR(NeuralNetworkArchitecture<T> architecture, OLMoASROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
OLMoASR(NeuralNetworkArchitecture<T> architecture, string modelPath, OLMoASROptions options)

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes the first 30 seconds of audio by greedy decoding after the prompt.
string DetectLanguage(Tensor<T> audio)