Paraformer<T>

Models & Types · AiDotNet.SpeechRecognition.AlibabaASR

Paraformer (Gao et al. 2022): a non-autoregressive recognizer whose CIF predictor counts the tokens and extracts one acoustic embedding per token, which a parallel decoder turns into the transcript in one pass.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → Paraformer<T>

Implements: ISpeechRecognizer<T>

For Beginners

Instead of writing the transcript word by word, Paraformer first predicts how many tokens the audio holds and gathers the stretch of audio belonging to each, then a decoder fills in every token at once. In training a “glancing” step shows the decoder some of the correct tokens so it learns how neighbouring tokens depend on each other.

How It Works

This is FunASR’s AISHELL-1 recipe (paraformer_conformer_12e_6d_2048_256), the configuration of the paper’s benchmark results: 80-band log-mel features, a Conformer encoder (4x convolution subsampling, width 256, 4 heads, feed-forward 2048, convolution kernel 15, 12 blocks), the depthwise CIF predictor (context 1 each side, tail weight 0.45), and a 6-block SAN decoder (self-attention without a causal mask, cross-attention and a ReLU feed-forward). A CTC layer on the encoder is weighted 0.3. Training minimizes 0.7 times the label-smoothed (0.1) decoder loss with the glancing sampler (λ = 0.4), 0.3 times the CTC loss, and the MAE between the target length and the predicted count; SpecAugment masks two frequency bands of up to 30 bins and two time spans (the recipe’s time warp is not applied). The recipe’s optimizer is Adam at 5e-4 with 30,000 warmup steps and gradient clipping at 5. The encoder’s blocks use rotary relative positions in place of the recipe’s Transformer-XL encoding (see ConformerAsrNetwork).

Example

var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.SpeechRecognition,
    inputSize: 16000, outputSize: 8404);
var model = new Paraformer<float>(architecture);
var text = model.Transcribe(waveform).Text;

Constructors

Constructor Summary
Paraformer(NeuralNetworkArchitecture<T> architecture, ParaformerOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
Paraformer(NeuralNetworkArchitecture<T> architecture, string modelPath, ParaformerOptions options)

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes audio: the CIF token count and acoustic embeddings, then one parallel decoder pass.
string DetectLanguage(Tensor<T> audio)