ParaformerLarge<T>

Models & Types · AiDotNet.SpeechRecognition.AlibabaASR

Paraformer-large (Gao et al. 2023, FunASR): the released Mandarin Paraformer, a SAN-M encoder and decoder around a CIF predictor.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → ParaformerLarge<T>

Implements: ISpeechRecognizer<T>

For Beginners

Paraformer-large is the production-size Paraformer: it predicts how many tokens the audio holds, gathers the audio belonging to each, and writes all of them in one decoder pass, which makes it many times faster than a recognizer that writes one token at a time.

How It Works

This is the release speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404 (about 220M parameters): 80-band Kaldi filterbanks, seven frames stacked every six and mean-variance normalized; a 50-block SAN-M encoder (SanmEncoder: width 512, 4 heads, feed-forward 2048, FSMN kernel 11); the convolutional CIF predictor (CifPredictorV2, context 1 each side, tail weight 0.45); and a 16-block SAN-M decoder (feed-forward, FSMN memory and cross-attention) with a final feed-forward block, over 8,404 characters. Every target ends with the end token (2), which the predictor counts. Training minimizes the length-normalized label-smoothed (0.1) decoder loss with the glancing sampler (λ = 0.75) plus the MAE of the token count; there is no CTC branch. The release’s SpecAugment on the stacked features is not applied. The optimizer is Adam at 5e-4 with 30,000 warmup steps and gradient clipping at 5.

Example

var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.SpeechRecognition,
    inputSize: 16000, outputSize: 8404);
var model = new ParaformerLarge<float>(architecture);
var text = model.Transcribe(waveform).Text;

Constructors

Constructor Summary
ParaformerLarge(NeuralNetworkArchitecture<T> architecture, ParaformerLargeOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
ParaformerLarge(NeuralNetworkArchitecture<T> architecture, string modelPath, ParaformerLargeOptions options)

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
ValueTuple<double[], double[]> Normalization { get; set; } The encoder’s CMVN shift and scale (FunASR’s am.mvn, read by ReadNormalization); identity until set.
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes audio: the CIF token count and acoustic embeddings, then one parallel decoder pass.
string DetectLanguage(Tensor<T> audio)