StreamingZipformer<T>

Models & Types · AiDotNet.SpeechRecognition.Streaming

Streaming Zipformer (Yao et al. 2024): a U-Net-shaped Transformer encoder whose stacks run at different frame rates, in its causal, chunk-wise form, with a stateless transducer decoder.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → StreamingZipformer<T>

Implements: ISpeechRecognizer<T>

For Beginners

Zipformer encodes speech at several time resolutions at once (fast near the input, slow in the middle), which makes it both accurate and efficient. This streaming version only looks at the current chunk of audio and a window of the past, so it can transcribe live.

How It Works

The network is icefall’s LibriSpeech Zipformer: Conv-Embed to 50 Hz, six stacks (2, 2, 3, 4, 3 and 2 blocks; widths 192, 256, 384, 512, 384 and 256; downsampling 1, 2, 4, 8, 4 and 2), BiasNorm, SwooshL and SwooshR, a stateless decoder of width 512 over the last two labels and a 512-wide joiner, with a 500-token BPE vocabulary. The streaming form attends within 32-frame chunks (at 50 Hz, reduced per stack) and 128 frames of left context, and its convolutions see only past frames; icefall’s chunk-causal convolution, which also lets the current chunk’s future frames in, is simplified here to a causal one. Training uses the transducer loss over SpecAugmented features (icefall trains with its pruned approximation of that loss).

Example

var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.SpeechRecognition,
    inputSize: 16000, outputSize: 500);
var model = new StreamingZipformer<float>(architecture);
var text = model.Transcribe(waveform).Text;

Constructors

Constructor Summary
StreamingZipformer(NeuralNetworkArchitecture<T> architecture, StreamingZipformerOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
StreamingZipformer(NeuralNetworkArchitecture<T> architecture, string modelPath, StreamingZipformerOptions options)

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes audio: the encoder and the transducer with greedy decoding.
string DetectLanguage(Tensor<T> audio)