SeACo<T>

Models & Types · AiDotNet.SpeechRecognition.AlibabaASR

SeACo-Paraformer (Shi et al. 2023): Paraformer-large with a bias encoder, bias decoder and bias output layer that bias recognition toward a hotword list given at inference.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → SeACo<T>

Implements: ISpeechRecognizer<T>

For Beginners

SeACo-Paraformer is a fast, non-autoregressive speech recognizer that you can hand a list of names or terms when you transcribe. A small extra network reads the list, checks at every output position whether one of those terms is being spoken, and if so steers the output toward it; elsewhere the recognizer’s own output is kept.

How It Works

The backbone is the released Paraformer-large (ParaformerLarge): a 50-block SAN-M encoder of width 512, the convolutional CIF predictor and a 16-block SAN-M decoder over 8,404 characters. The release’s CifPredictorV3 adds an upsampled timestamp branch to the same CIF path; it is not built, since this model emits no timestamps. The bias modules (section 3.1, with the sizes of section 4.2) are a two-layer unidirectional LSTM of width 512 over the hotwords’ ASR-decoder embeddings, a SAN-M bias decoder of four blocks with 1,024 hidden units (FSMN kernel 21, the release’s) reading the CIF acoustic embeddings and the ASR decoder’s hidden states over the hotword representations, and an output layer over the vocabulary plus the appended no-bias class ‘#’. (The release’s configuration declares four blocks, but its decoder class builds att_layer_num, default 6; the paper’s four are used.)

Training follows the paper’s two stages (TrainingStage). The backbone stage minimizes Paraformer-large’s loss. The bias stage freezes the backbone, samples hotwords from the batch’s transcripts (a share r_b = 0.75 of batches, r_u = 0.75 of their utterances, 2 to 8 tokens, plus the default hotword, the start token) and minimizes the length-normalized label-smoothed (0.1) loss of the bias logits against the transcript with every non-hotword position replaced by ‘#’. At inference the merge of Equation 5 keeps the ASR probabilities where the bias decoder predicts ‘#’ and uses λ P_bias + (1 − λ) P_ASR elsewhere (λ = 1); with more than k = 50 hotwords, attention score filtering keeps the k the bias decoder’s last layer attends to most. The optimizer is Adam at 5e-4 with 30,000 warmup steps and gradient clipping at 5.

Example

var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.SpeechRecognition,
    inputSize: 16000, outputSize: 8404);
var model = new SeACo<float>(architecture);
var text = model.Transcribe(waveform, new[] { "达摩院", "魔搭" }).Text;

Constructors

Constructor Summary
SeACo(NeuralNetworkArchitecture<T> architecture, SeACoOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
SeACo(NeuralNetworkArchitecture<T> architecture, string modelPath, SeACoOptions options)

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
ValueTuple<double[], double[]> Normalization { get; set; } The encoder’s CMVN shift and scale (FunASR’s am.mvn, read by ReadNormalization); identity until set.
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input)
TranscriptionResult<T> Transcribe(Tensor<T> audio, IReadOnlyList<string> hotwords, string language, bool? includeTimestamps) Transcribes audio biased toward hotwords (names, terms, product names): the bias decoder reads the hotword list, and Equation 5 merges its probabilities with the backbone’s wherever it detects a hotword.
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes audio without hotwords: the backbone alone, as Equation 5 prescribes when none is given.
string DetectLanguage(Tensor<T> audio)