WhisperModel<T>

Models & Types · AiDotNet.Audio.Whisper

Whisper automatic speech recognition model for transcribing audio to text.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → WhisperModel<T>

Implements: IPaperOptimizerVariant, ISpeechRecognizer<T>

For Beginners

Whisper converts spoken audio into text. It works by:

  1. Converting audio to a mel spectrogram (visual representation of sound)
  2. Processing through an encoder neural network
  3. Generating text tokens through a decoder neural network

Two ways to use this class:

  1. ONNX Mode: Load pretrained models for fast inference
  2. Native Mode: Train your own speech recognition model from scratch

ONNX Mode Example:

Training Mode Example:

How It Works

Whisper is a state-of-the-art speech recognition model by OpenAI that can:

  • Transcribe speech in 99+ languages
  • Translate non-English speech to English
  • Detect the spoken language automatically
  • Handle noisy audio and accents well

Constructors

Constructor Summary
WhisperModel(NeuralNetworkArchitecture<T> architecture, WhisperOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a Whisper network for training from scratch using native layers.
WhisperModel(NeuralNetworkArchitecture<T> architecture, string encoderPath, string decoderPath, WhisperOptions options, OnnxModelOptions onnxOptions) Creates a Whisper network using pretrained ONNX models.

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; } Gets the list of languages supported by this model.
WhisperModelSize ModelSize { get; } Gets the model size variant.
bool IsReady { get; } Gets whether the model is ready for inference.
bool SupportsStreaming { get; } Gets whether this model supports real-time streaming transcription.
bool SupportsWordTimestamps { get; } Gets whether this model can identify timestamps for each word.
bool Translate { get; } Gets whether translation to English is enabled.
int MaxAudioLengthSeconds { get; } Gets the maximum audio length in seconds.
string Language { get; } Gets the target language for transcription.
string PaperOptimizerVariant { get; } The size this instance was built at, keying the paper recipe for it.

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) Gets language detection probabilities for the audio.
IStreamingTranscriptionSession<T> StartStreamingSession(string language) Starts a streaming transcription session.
ModelMetadata<T> GetModelMetadata() Gets metadata about the model.
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) Transcribes audio to text asynchronously.
Tensor<T> ForwardForTraining(Tensor<T> input) Runs paper-correct teacher-forced native training: waveform to log-mel, audio encoder, shifted transcript tokens, decoder vocabulary logits.
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes audio to text.
string DetectLanguage(Tensor<T> audio) Detects the language spoken in the audio.