Paraformer<T>
Models & Types · AiDotNet.SpeechRecognition.AlibabaASR
Paraformer (Gao et al. 2022): a non-autoregressive recognizer whose CIF predictor counts the tokens and extracts one acoustic embedding per token, which a parallel decoder turns into the transcript in one pass.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → Paraformer<T>
Implements: ISpeechRecognizer<T>
For Beginners
Instead of writing the transcript word by word, Paraformer first predicts how many tokens the audio holds and gathers the stretch of audio belonging to each, then a decoder fills in every token at once. In training a “glancing” step shows the decoder some of the correct tokens so it learns how neighbouring tokens depend on each other.
How It Works
This is FunASR’s AISHELL-1 recipe (paraformer_conformer_12e_6d_2048_256), the configuration of the paper’s
benchmark results: 80-band log-mel features, a Conformer encoder (4x convolution subsampling, width 256, 4 heads,
feed-forward 2048, convolution kernel 15, 12 blocks), the depthwise CIF predictor (context 1 each side, tail weight
0.45), and a 6-block SAN decoder (self-attention without a causal mask, cross-attention and a ReLU feed-forward). A CTC
layer on the encoder is weighted 0.3. Training minimizes 0.7 times the label-smoothed (0.1) decoder loss with the
glancing sampler (λ = 0.4), 0.3 times the CTC loss, and the MAE between the target length and the predicted count;
SpecAugment masks two frequency bands of up to 30 bins and two time spans (the recipe’s time warp is not applied).
The recipe’s optimizer is Adam at 5e-4 with 30,000 warmup steps and gradient clipping at 5. The encoder’s blocks use
rotary relative positions in place of the recipe’s Transformer-XL encoding (see ConformerAsrNetwork).
Example
var architecture = new NeuralNetworkArchitecture<float>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.SpeechRecognition,
inputSize: 16000, outputSize: 8404);
var model = new Paraformer<float>(architecture);
var text = model.Transcribe(waveform).Text;
Constructors
| Constructor | Summary |
|---|---|
Paraformer(NeuralNetworkArchitecture<T> architecture, ParaformerOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
Paraformer(NeuralNetworkArchitecture<T> architecture, string modelPath, ParaformerOptions options) |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
|
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
Transcribes audio: the CIF token count and acoustic embeddings, then one parallel decoder pass. |
string DetectLanguage(Tensor<T> audio) |