SeACo<T>
Models & Types · AiDotNet.SpeechRecognition.AlibabaASR
SeACo-Paraformer (Shi et al. 2023): Paraformer-large with a bias encoder, bias decoder and bias output layer that bias recognition toward a hotword list given at inference.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → SeACo<T>
Implements: ISpeechRecognizer<T>
For Beginners
SeACo-Paraformer is a fast, non-autoregressive speech recognizer that you can hand a list of names or terms when you transcribe. A small extra network reads the list, checks at every output position whether one of those terms is being spoken, and if so steers the output toward it; elsewhere the recognizer’s own output is kept.
How It Works
The backbone is the released Paraformer-large (ParaformerLarge): a 50-block SAN-M encoder of width
512, the convolutional CIF predictor and a 16-block SAN-M decoder over 8,404 characters. The release’s
CifPredictorV3 adds an upsampled timestamp branch to the same CIF path; it is not built, since this model emits
no timestamps. The bias modules (section 3.1, with the sizes of section 4.2) are a two-layer unidirectional LSTM of
width 512 over the hotwords’ ASR-decoder embeddings, a SAN-M bias decoder of four blocks with 1,024 hidden units (FSMN
kernel 21, the release’s) reading the CIF acoustic embeddings and the ASR decoder’s hidden states over the hotword
representations, and an output layer over the vocabulary plus the appended no-bias class ‘#’. (The release’s
configuration declares four blocks, but its decoder class builds att_layer_num, default 6; the paper’s four are
used.)
Training follows the paper’s two stages (TrainingStage). The backbone stage minimizes
Paraformer-large’s loss. The bias stage freezes the backbone, samples hotwords from the batch’s transcripts (a share
r_b = 0.75 of batches, r_u = 0.75 of their utterances, 2 to 8 tokens, plus the default hotword, the start token) and
minimizes the length-normalized label-smoothed (0.1) loss of the bias logits against the transcript with every
non-hotword position replaced by ‘#’. At inference the merge of Equation 5 keeps the ASR probabilities where the bias
decoder predicts ‘#’ and uses λ P_bias + (1 − λ) P_ASR elsewhere (λ = 1); with more than k = 50 hotwords, attention
score filtering keeps the k the bias decoder’s last layer attends to most. The optimizer is Adam at 5e-4 with 30,000
warmup steps and gradient clipping at 5.
Example
var architecture = new NeuralNetworkArchitecture<float>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.SpeechRecognition,
inputSize: 16000, outputSize: 8404);
var model = new SeACo<float>(architecture);
var text = model.Transcribe(waveform, new[] { "达摩院", "魔搭" }).Text;
Constructors
| Constructor | Summary |
|---|---|
SeACo(NeuralNetworkArchitecture<T> architecture, SeACoOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
SeACo(NeuralNetworkArchitecture<T> architecture, string modelPath, SeACoOptions options) |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
ValueTuple<double[], double[]> Normalization { get; set; } |
The encoder’s CMVN shift and scale (FunASR’s am.mvn, read by ReadNormalization); identity until set. |
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
|
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, IReadOnlyList<string> hotwords, string language, bool? includeTimestamps) |
Transcribes audio biased toward hotwords (names, terms, product names): the bias decoder reads the hotword list, and Equation 5 merges its probabilities with the backbone’s wherever it detects a hotword. |
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
Transcribes audio without hotwords: the backbone alone, as Equation 5 prescribes when none is given. |
string DetectLanguage(Tensor<T> audio) |