CanaryQwen<T>
Models & Types · AiDotNet.SpeechRecognition.NeMo
Canary-Qwen-2.5B (NVIDIA, 2025): a speech-augmented language model (SALM, Chen et al. 2023) built from canary-1b-flash’s Fast Conformer encoder, a linear projection and Qwen3-1.7B with LoRA.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → CanaryQwen<T>
Implements: ISpeechRecognizer<T>
For Beginners
Canary-Qwen listens with Canary’s speech encoder, maps each 80 ms of what it hears into the language model’s word space with one linear layer, and asks Qwen3 “Transcribe the following:” with that audio in the chat message. Qwen3 then writes the transcript.
How It Works
The release (config.json and the model card) fixes the network: NeMo’s ConformerEncoder at width 1,024
with 32 blocks, 8 heads, kernel 9 and 8x depthwise-separable striding over 128 log-mel bands; an identity modality
adapter; AudioPerceptionModule.proj, a linear map to 2,048; and Qwen3-1.7B with LoRA of rank 128 (alpha 256,
dropout 0.01) on q_proj and v_proj. The prompt is NeMo’s qwen format,
<|im_start|>user\nTranscribe the following: <audio><|im_end|>\n<|im_start|>assistant\n,
and the answer ends with <|im_end|>. Training freezes the LLM (freeze_params) and updates the
encoder, the projection and LoRA, with AdamW (learning rate 5e-4, betas 0.9 and 0.98, weight decay 1e-3) under cosine
annealing after 1,000 warm-up steps to 1e-6. A model built here starts from random weights and a frozen random LLM
cannot fit a transcript, so the default updates every parameter; the release’s scope is
EncoderAdapterAndLora.
Example
var architecture = new NeuralNetworkArchitecture<float>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.SpeechRecognition,
inputSize: 160000, outputSize: 151936);
var model = new CanaryQwen<float>(architecture);
var text = model.Transcribe(waveform).Text;
Constructors
| Constructor | Summary |
|---|---|
CanaryQwen(NeuralNetworkArchitecture<T> architecture, CanaryQwenOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
CanaryQwen(NeuralNetworkArchitecture<T> architecture, string modelPath, CanaryQwenOptions options) |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
|
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
Transcribes audio: the LLM’s greedy answer to the instruction, at most MaxTextLength tokens. |
string DetectLanguage(Tensor<T> audio) |