SenseVoiceLarge<T>

Models & Types · AiDotNet.SpeechRecognition.AlibabaASR

SenseVoice-Large (FunAudioLLM, 2024): an autoregressive encoder-decoder for multilingual recognition, language identification, speech emotion recognition and audio event detection, its tasks selected by decoder input tokens.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → SenseVoiceLarge<T>

Implements: ISpeechRecognizer<T>

For Beginners

SenseVoice-Large listens to up to 30 seconds of audio and writes the transcript one token at a time. Before the transcript you can ask it to name the language, the speaker’s emotion and the audio events, by putting the matching task tokens at the start of what the decoder reads.

How It Works

The report (section 2.1) describes SenseVoice-Large as an autoregressive encoder-decoder that, “similar to Whisper”, specifies tasks by input tokens to the decoder (LID, SER and AED), with 1,587M parameters; the CosyVoice report builds its S3 tokenizer from the first six layers of this encoder, which the released tokenizer shows to be Whisper-large’s (128 mel bands, width 1,280, 20 heads). The network is therefore openai-whisper’s (OpenAiWhisperNetwork) at the large geometry: a 30-second log-mel window, two convolutions, 32 pre-norm encoder layers, and 32 decoder layers with cross-attention, tied to the token embedding. The decoder reads the start of transcript followed by the selected task tokens and predicts the task outputs and the transcript, closed by end of text; training minimizes their teacher-forced cross-entropy. The vocabulary is unpublished: its size is configurable, with the five special tokens at its top (see VocabSize). The report states no optimizer; AdamW at 1e-4 is the default.

Example

var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.SpeechRecognition,
    inputSize: 480000, outputSize: 51866);
var model = new SenseVoiceLarge<float>(architecture);
var text = model.Transcribe(waveform).Text;

Constructors

Constructor Summary
SenseVoiceLarge(NeuralNetworkArchitecture<T> architecture, SenseVoiceLargeOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
SenseVoiceLarge(NeuralNetworkArchitecture<T> architecture, string modelPath, SenseVoiceLargeOptions options)

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes the first 30 seconds of audio by greedy decoding after the prompt.
string DetectLanguage(Tensor<T> audio)