SenseVoice<T>

Models & Types · AiDotNet.SpeechRecognition.AlibabaASR

SenseVoice-Small (An et al. 2024, FunAudioLLM §2.2): a non-autoregressive SAN-M encoder that recognizes speech and classifies its language, emotion and audio events from four task queries prepended to the features.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → SenseVoice<T>

Implements: ISpeechRecognizer<T>

For Beginners

SenseVoice-Small turns 16 kHz audio into text with one encoder pass, no decoder. Four extra “question” vectors are put in front of the audio frames; the encoder’s answers at those four positions are the spoken language, the emotion, the audio event and whether the text is normalized, and the remaining positions spell the transcript with CTC.

How It Works

The front end is the release’s: 80-band Kaldi filterbanks of the waveform, seven frames stacked every six (low frame rate) and mean-variance normalized, giving 560 features per 60 ms. Four query embeddings of the same width — the language (auto 0, zh 3, en 4, yue 7, ja 11, ko 12, nospeech 13), the emotion (1), the event (2) and the text normalization (with 14, without 15) — are prepended in that order. The encoder is SanmEncoder: 50 SAN-M blocks (self-attention plus an FSMN memory, width 512, 4 heads, feed-forward 2048, kernel 11), a norm, 20 more blocks and a norm. A linear layer (ctc_lo) maps every position onto the 25,055-token vocabulary: positions 4 onward are the CTC transcript and the first four the rich tokens. Training minimizes the CTC loss of the transcript plus, when the targets carry the four rich tokens first (the release’s layout), their cross-entropy through the same layer.

Example

var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.SpeechRecognition,
    inputSize: 16000, outputSize: 25055);
var model = new SenseVoice<float>(architecture);
var text = model.Transcribe(waveform).Text;

Constructors

Constructor Summary
SenseVoice(NeuralNetworkArchitecture<T> architecture, SenseVoiceOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
SenseVoice(NeuralNetworkArchitecture<T> architecture, string modelPath, SenseVoiceOptions options)

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
ValueTuple<double[], double[]> Normalization { get; set; } The encoder’s CMVN shift and scale (FunASR’s am.mvn, read by ReadNormalization); identity until set.
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes audio: the CTC transcript of the frames after the four task queries, decoded greedily.
int[] RichTokenIds(Tensor<T> audio) The most likely vocabulary token at each of the four query positions (language, emotion, event, text normalization), the language query auto.
string DetectLanguage(Tensor<T> audio) The configured language: the vocabulary’s language tokens belong to the tokenizer, so the rich output (RichTokenIds) is returned as token ids rather than named here.