SenseVoice<T>
Models & Types · AiDotNet.SpeechRecognition.AlibabaASR
SenseVoice-Small (An et al. 2024, FunAudioLLM §2.2): a non-autoregressive SAN-M encoder that recognizes speech and classifies its language, emotion and audio events from four task queries prepended to the features.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → SenseVoice<T>
Implements: ISpeechRecognizer<T>
For Beginners
SenseVoice-Small turns 16 kHz audio into text with one encoder pass, no decoder. Four extra “question” vectors are put in front of the audio frames; the encoder’s answers at those four positions are the spoken language, the emotion, the audio event and whether the text is normalized, and the remaining positions spell the transcript with CTC.
How It Works
The front end is the release’s: 80-band Kaldi filterbanks of the waveform, seven frames stacked every six (low frame
rate) and mean-variance normalized, giving 560 features per 60 ms. Four query embeddings of the same width — the
language (auto 0, zh 3, en 4, yue 7, ja 11, ko 12, nospeech 13), the emotion (1), the event (2) and the text
normalization (with 14, without 15) — are prepended in that order. The encoder is
SanmEncoder: 50 SAN-M blocks (self-attention plus an FSMN memory, width 512, 4 heads, feed-forward
2048, kernel 11), a norm, 20 more blocks and a norm. A linear layer (ctc_lo) maps every position onto the
25,055-token vocabulary: positions 4 onward are the CTC transcript and the first four the rich tokens. Training
minimizes the CTC loss of the transcript plus, when the targets carry the four rich tokens first (the release’s
layout), their cross-entropy through the same layer.
Example
var architecture = new NeuralNetworkArchitecture<float>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.SpeechRecognition,
inputSize: 16000, outputSize: 25055);
var model = new SenseVoice<float>(architecture);
var text = model.Transcribe(waveform).Text;
Constructors
| Constructor | Summary |
|---|---|
SenseVoice(NeuralNetworkArchitecture<T> architecture, SenseVoiceOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
SenseVoice(NeuralNetworkArchitecture<T> architecture, string modelPath, SenseVoiceOptions options) |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
ValueTuple<double[], double[]> Normalization { get; set; } |
The encoder’s CMVN shift and scale (FunASR’s am.mvn, read by ReadNormalization); identity until set. |
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
|
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
Transcribes audio: the CTC transcript of the frames after the four task queries, decoded greedily. |
int[] RichTokenIds(Tensor<T> audio) |
The most likely vocabulary token at each of the four query positions (language, emotion, event, text normalization), the language query auto. |
string DetectLanguage(Tensor<T> audio) |
The configured language: the vocabulary’s language tokens belong to the tokenizer, so the rich output (RichTokenIds) is returned as token ids rather than named here. |