WhisperModel<T>
Models & Types · AiDotNet.Audio.Whisper
Whisper automatic speech recognition model for transcribing audio to text.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → WhisperModel<T>
Implements: IPaperOptimizerVariant, ISpeechRecognizer<T>
For Beginners
Whisper converts spoken audio into text. It works by:
- Converting audio to a mel spectrogram (visual representation of sound)
- Processing through an encoder neural network
- Generating text tokens through a decoder neural network
Two ways to use this class:
- ONNX Mode: Load pretrained models for fast inference
- Native Mode: Train your own speech recognition model from scratch
ONNX Mode Example:
Training Mode Example:
How It Works
Whisper is a state-of-the-art speech recognition model by OpenAI that can:
- Transcribe speech in 99+ languages
- Translate non-English speech to English
- Detect the spoken language automatically
- Handle noisy audio and accents well
Constructors
| Constructor | Summary |
|---|---|
WhisperModel(NeuralNetworkArchitecture<T> architecture, WhisperOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
Creates a Whisper network for training from scratch using native layers. |
WhisperModel(NeuralNetworkArchitecture<T> architecture, string encoderPath, string decoderPath, WhisperOptions options, OnnxModelOptions onnxOptions) |
Creates a Whisper network using pretrained ONNX models. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
Gets the list of languages supported by this model. |
WhisperModelSize ModelSize { get; } |
Gets the model size variant. |
bool IsReady { get; } |
Gets whether the model is ready for inference. |
bool SupportsStreaming { get; } |
Gets whether this model supports real-time streaming transcription. |
bool SupportsWordTimestamps { get; } |
Gets whether this model can identify timestamps for each word. |
bool Translate { get; } |
Gets whether translation to English is enabled. |
int MaxAudioLengthSeconds { get; } |
Gets the maximum audio length in seconds. |
string Language { get; } |
Gets the target language for transcription. |
string PaperOptimizerVariant { get; } |
The size this instance was built at, keying the paper recipe for it. |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
Gets language detection probabilities for the audio. |
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
Starts a streaming transcription session. |
ModelMetadata<T> GetModelMetadata() |
Gets metadata about the model. |
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
Transcribes audio to text asynchronously. |
Tensor<T> ForwardForTraining(Tensor<T> input) |
Runs paper-correct teacher-forced native training: waveform to log-mel, audio encoder, shifted transcript tokens, decoder vocabulary logits. |
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
Transcribes audio to text. |
string DetectLanguage(Tensor<T> audio) |
Detects the language spoken in the audio. |