SenseVoiceLarge<T>
Models & Types · AiDotNet.SpeechRecognition.AlibabaASR
SenseVoice-Large (FunAudioLLM, 2024): an autoregressive encoder-decoder for multilingual recognition, language identification, speech emotion recognition and audio event detection, its tasks selected by decoder input tokens.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → SenseVoiceLarge<T>
Implements: ISpeechRecognizer<T>
For Beginners
SenseVoice-Large listens to up to 30 seconds of audio and writes the transcript one token at a time. Before the transcript you can ask it to name the language, the speaker’s emotion and the audio events, by putting the matching task tokens at the start of what the decoder reads.
How It Works
The report (section 2.1) describes SenseVoice-Large as an autoregressive encoder-decoder that, “similar to Whisper”,
specifies tasks by input tokens to the decoder (LID, SER and AED), with 1,587M parameters; the CosyVoice report builds
its S3 tokenizer from the first six layers of this encoder, which the released tokenizer shows to be Whisper-large’s
(128 mel bands, width 1,280, 20 heads). The network is therefore openai-whisper’s
(OpenAiWhisperNetwork) at the large geometry: a 30-second log-mel window, two convolutions, 32
pre-norm encoder layers, and 32 decoder layers with cross-attention, tied to the token embedding. The decoder reads the
start of transcript followed by the selected task tokens and predicts the task outputs and the transcript, closed by
end of text; training minimizes their teacher-forced cross-entropy. The vocabulary is unpublished: its size is
configurable, with the five special tokens at its top (see VocabSize). The report
states no optimizer; AdamW at 1e-4 is the default.
Example
var architecture = new NeuralNetworkArchitecture<float>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.SpeechRecognition,
inputSize: 480000, outputSize: 51866);
var model = new SenseVoiceLarge<float>(architecture);
var text = model.Transcribe(waveform).Text;
Constructors
| Constructor | Summary |
|---|---|
SenseVoiceLarge(NeuralNetworkArchitecture<T> architecture, SenseVoiceLargeOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
SenseVoiceLarge(NeuralNetworkArchitecture<T> architecture, string modelPath, SenseVoiceLargeOptions options) |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
|
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
Transcribes the first 30 seconds of audio by greedy decoding after the prompt. |
string DetectLanguage(Tensor<T> audio) |