OWSM<T>

Models & Types · AiDotNet.SpeechRecognition.Multilingual

OWSM v3.1 (Peng et al. 2024): an open Whisper-style speech model with an E-Branchformer encoder.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → OWSM<T>

Implements: ISpeechRecognizer<T>

For Beginners

OWSM v3.1 turns 16 kHz audio into an 80-band log-mel spectrogram, shortens it 4x with two convolutions, and encodes it with Conformer blocks, each mixing self-attention (long-range context) with a convolution (local patterns). A linear layer gives every frame a probability for each token; collapsing repeats and blanks gives the transcript.

How It Works

The defaults are OWSM v3.1 medium (Table 1 and its released configuration): an 18-block E-Branchformer encoder, width 1024, 16 heads, 4096 gating-MLP and feed-forward units, 31-wide gating and merge convolutions and sinusoidal absolute positions (section 2.1), an 18-layer Transformer decoder with 4096 feed-forward units, CTC weight 0.3, label smoothing 0.1 and the 50,002-token vocabulary. The decoder here reads only the sentence boundary token, not OWSM’s language, task and timestamp prompt. SpecAugment masks the features in training (section 3.1). Training minimizes the joint CTC/attention loss (Watanabe et al. 2017); transcription decodes the attention decoder greedily.

Example

var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.SpeechRecognition,
    inputSize: 16000, outputSize: 34);
var model = new OWSM<float>(architecture);
var text = model.Transcribe(waveform).Text;

Constructors

Constructor Summary
OWSM(NeuralNetworkArchitecture<T> architecture, OWSMOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
OWSM(NeuralNetworkArchitecture<T> architecture, string modelPath, OWSMOptions options)

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes audio: the encoder and the attention decoder with greedy decoding.
string DetectLanguage(Tensor<T> audio)