ParaformerLarge<T>
Models & Types · AiDotNet.SpeechRecognition.AlibabaASR
Paraformer-large (Gao et al. 2023, FunASR): the released Mandarin Paraformer, a SAN-M encoder and decoder around a CIF predictor.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → ParaformerLarge<T>
Implements: ISpeechRecognizer<T>
For Beginners
Paraformer-large is the production-size Paraformer: it predicts how many tokens the audio holds, gathers the audio belonging to each, and writes all of them in one decoder pass, which makes it many times faster than a recognizer that writes one token at a time.
How It Works
This is the release speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404 (about 220M parameters): 80-band
Kaldi filterbanks, seven frames stacked every six and mean-variance normalized; a 50-block SAN-M encoder
(SanmEncoder: width 512, 4 heads, feed-forward 2048, FSMN kernel 11); the convolutional CIF predictor
(CifPredictorV2, context 1 each side, tail weight 0.45); and a 16-block SAN-M decoder (feed-forward, FSMN memory
and cross-attention) with a final feed-forward block, over 8,404 characters. Every target ends with the end token
(2), which the predictor counts. Training minimizes the length-normalized label-smoothed (0.1) decoder loss with the
glancing sampler (λ = 0.75) plus the MAE of the token count; there is no CTC branch. The release’s SpecAugment on the
stacked features is not applied. The optimizer is Adam at 5e-4 with 30,000 warmup steps and gradient clipping at 5.
Example
var architecture = new NeuralNetworkArchitecture<float>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.SpeechRecognition,
inputSize: 16000, outputSize: 8404);
var model = new ParaformerLarge<float>(architecture);
var text = model.Transcribe(waveform).Text;
Constructors
| Constructor | Summary |
|---|---|
ParaformerLarge(NeuralNetworkArchitecture<T> architecture, ParaformerLargeOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
ParaformerLarge(NeuralNetworkArchitecture<T> architecture, string modelPath, ParaformerLargeOptions options) |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
ValueTuple<double[], double[]> Normalization { get; set; } |
The encoder’s CMVN shift and scale (FunASR’s am.mvn, read by ReadNormalization); identity until set. |
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
|
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
Transcribes audio: the CIF token count and acoustic embeddings, then one parallel decoder pass. |
string DetectLanguage(Tensor<T> audio) |