Zipformer<T>

Models & Types · AiDotNet.Audio.SpeechRecognition

Zipformer speech recognition model (Yao et al., 2023, Next-gen Kaldi).

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → Zipformer<T>

Implements: ISpeechRecognizer<T>

For Beginners

Zipformer is an improved version of the Conformer that’s both faster and more accurate. It processes speech at different “zoom levels”:

  • Some layers look at fine details (individual sounds at full resolution)
  • Middle layers zoom out (every 2nd, 4th, or 8th frame) for bigger-picture patterns
  • Then it zooms back in to combine everything

This U-Net-like structure (like looking through a microscope at different magnifications) makes it one of the most efficient speech encoders available today.

Usage:

How It Works

Zipformer (Yao et al., 2023) is a more efficient variant of the Conformer with a U-Net-like structure that processes speech at different temporal resolutions. It uses BiasNorm instead of LayerNorm, SwooshR/SwooshL activations, and temporal downsampling across encoder stacks. This achieves better accuracy with fewer parameters than standard Conformer, with WER 2.0%/4.4% on LibriSpeech test-clean/other without an external language model.

Constructors

Constructor Summary
Zipformer(NeuralNetworkArchitecture<T> architecture, ZipformerOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a Zipformer model in native training mode.
Zipformer(NeuralNetworkArchitecture<T> architecture, string modelPath, ZipformerOptions options) Creates a Zipformer model in native training mode.

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps)
string DetectLanguage(Tensor<T> audio)