FunASRNano<T>

Models & Types · AiDotNet.SpeechRecognition.AlibabaASR

Fun-ASR-nano (Tongyi Lab, 2025): an LLM-based recognizer of a SenseVoice-Small audio encoder, a Transformer adaptor and the Qwen3-0.6B decoder, with a CTC decoder whose first-pass hypothesis drives hotword retrieval.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → FunASRNano<T>

Implements: ISpeechRecognizer<T>

For Beginners

Fun-ASR-nano listens to speech, turns it into vectors a language model can read, and lets the language model (Qwen3-0.6B) write the transcript as the answer to a chat prompt that asks for one. A separate, faster CTC decoder produces a rough first transcript that is used to look up the hotwords to put in the prompt.

How It Works

The report (§3, Figure 2) gives four parts: an audio encoder (0.2B parameters for nano), an adaptor of two Transformer encoder layers, a CTC decoder on the audio encoder, and the LLM decoder. The release (Fun-ASR-Nano-2512’s config and FunASRNano model) fixes them: SenseVoice-Small’s encoder (50 + 20 SAN-M blocks of width 512 over 80-band filterbanks stacked seven every six frames); the adaptor, a linear-ReLU-linear projection to the LLM width and two Transformer encoder layers; Qwen3-0.6B, reading the adaptor’s output in place of the speech inside a ChatML prompt (“You are a helpful assistant.” as system, the instruction “语音转写:” before the speech); and the CTC decoder, five Transformer encoder layers and a linear head over a 60,515-token vocabulary on the detached encoder output. The LLM’s loss is the transcript’s cross-entropy after the prompt. Training follows the report’s §4.2 stages (TrainingStage): only the modules the stage names are updated, LoRA adapters (rank 16) stand in for the LLM, and the CTC stage trains the CTC decoder alone with targets in its own vocabulary. Those stages presuppose a pretrained Qwen3, so the default is joint training of every parameter on the LLM’s loss. Without Qwen’s 151,936-token vocabulary (a smaller VocabSize) the prompt has no text: the LLM reads the speech alone and its last token ends the answer.

Example

var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.SpeechRecognition,
    inputSize: 480000, outputSize: 151936);
var model = new FunASRNano<float>(architecture);
var text = model.Transcribe(waveform).Text;

Constructors

Constructor Summary
FunASRNano(NeuralNetworkArchitecture<T> architecture, FunASRNanoOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
FunASRNano(NeuralNetworkArchitecture<T> architecture, string modelPath, FunASRNanoOptions options)

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes audio: the LLM’s greedy answer to the transcription prompt, with Hotwords in its context section.
int[] FirstPassTokens(Tensor<T> audio) The CTC decoder’s first-pass hypothesis (ids in the CTC vocabulary), which the report uses to retrieve the hotwords for the prompt.
string DetectLanguage(Tensor<T> audio)