ISpeechRecognizer<T>

Interfaces · AiDotNet.Interfaces

Interface for speech recognition models that transcribe audio to text (ASR - Automatic Speech Recognition).

Implements: IFeatureAware, IFullModel<T, Tensor<T>, Tensor<T>>, IGradientComputable<T, Tensor<T>, Tensor<T>>

Known implementations: AVHuBERT<T>, AWSTranscribe<T>, AssemblyAIUniversal2<T>, AudioPaLM<T>, AzureSpeechSTT<T>, BESTRQ<T>, Branchformer<T>, CIFDecoder<T>, CIFEncoder<T>, CTCDecoder<T>, CTCSegmentation<T>, CanaryFlash<T>, CanaryQwen<T>, Canary<T>, Chirp2<T>, Chirp3<T>, Chirp<T>, CodeSwitchingASR<T>, ConformerCTC<T>, ConformerTransducer<T>, Conformer<T>, ContextNet<T>, ConvTransformer<T>, Data2VecASR<T>, DeepgramNova2<T>, DistilWhisper<T>, EBranchformer<T>, ESPnetASR<T>, EfficientConformer<T>, EmformerRNNT<T>, FastConformer<T>, FastEmit<T>, FasterWhisper<T>, FireRedASRLLM<T>, FireRedASR<T>, FunASRNano<T>, GladiaASR<T>, GoogleSpeechV2<T>, GraniteSpeech<T>, GroqWhisper<T>, HuBERTASR<T>, InterCTC<T>, KeywordSpotting<T>, KotobaWhisper<T>, MMS<T>, MedicalASR<T>, MoonshineBase<T>, Moonshine<T>, Moshi<T>, NeMoCitrinet<T>, NeMoMultitask<T>, NemotronSpeech<T>, NoiseRobustASR<T>, OLMoASR<T>, OWSM<T>, OmnilangualASR<T>, ParaformerLarge<T>, Paraformer<T>, ParakeetCTC<T>, ParakeetRNNT<T> (+40 more)

For Beginners

Speech recognition is like having a transcriptionist listen to audio and type out what they hear.

How speech recognition works:

  1. Audio is converted to features (spectrograms or mel-spectrograms)
  2. The model processes these features to identify speech patterns
  3. Patterns are decoded into words and sentences

Common use cases:

  • Voice assistants (Siri, Alexa, Google Assistant)
  • Video/podcast transcription
  • Real-time captioning for accessibility
  • Voice typing and dictation

Key challenges:

  • Different accents and speaking styles
  • Background noise and multiple speakers
  • Domain-specific vocabulary (medical, legal terms)

How It Works

Speech recognition models convert spoken audio into written text. They analyze audio waveforms or spectrograms to identify phonemes, words, and sentences. Modern speech recognition uses encoder-decoder architectures (like Whisper) or CTC-based models.

This interface extends IFullModel for Tensor-based audio processing.

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; } Gets the list of languages supported by this model.
bool IsOnnxMode { get; } Gets whether this model is running in ONNX inference mode.
bool SupportsStreaming { get; } Gets whether this model supports real-time streaming transcription.
bool SupportsWordTimestamps { get; } Gets whether this model can identify timestamps for each word.
int SampleRate { get; } Gets the sample rate expected by this model.

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) Gets language detection probabilities for the audio.
IStreamingTranscriptionSession<T> StartStreamingSession(string language) Starts a streaming transcription session.
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) Transcribes audio to text asynchronously.
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes audio to text.
string DetectLanguage(Tensor<T> audio) Detects the language spoken in the audio.