Canary<T>

Models & Types · AiDotNet.Audio.SpeechRecognition

Canary multilingual speech recognition and translation model from NVIDIA.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → Canary<T>

Implements: ISpeechRecognizer<T>

For Beginners

Canary is like a multilingual transcription assistant. It can listen to speech in many languages and either transcribe it (write down what was said) or translate it into another language, all with a single model. You tell it what you want via a prompt.

Usage:

How It Works

Canary (NVIDIA, 2024) is a multilingual ASR/ST model based on the Fast Conformer encoder with a multi-task decoder. It supports transcription and translation across many languages using a single unified architecture with task-specific prompting, achieving strong WER scores across English, German, Spanish, and French.

Constructors

Constructor Summary
Canary(NeuralNetworkArchitecture<T> architecture, CanaryOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a Canary model in native training mode.
Canary(NeuralNetworkArchitecture<T> architecture, string modelPath, CanaryOptions options) Creates a Canary model in native training mode.

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
bool IsOnnxMode { get; }
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }
int SampleRate { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps)
string DetectLanguage(Tensor<T> audio)