Chirp<T>

Models & Types · AiDotNet.SpeechRecognition.Multilingual

Chirp: Google Cloud’s first-generation Chirp recognizer, the 2B-parameter Universal Speech Model (USM) with a CTC layer, served through Speech-to-Text V2.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → Chirp<T>

Implements: ISpeechRecognizer<T>

For Beginners

Chirp is the speech model Google serves in its cloud for 100+ languages. It listens to 16 kHz audio turned into a 128-band spectrogram, encodes it with a large Conformer, and outputs for every 40 ms frame a probability for each token; collapsing repeats and blanks gives the transcript.

How It Works

References:

  • Paper: “Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages” (Zhang et al., Google, 2023)
  • Model: “Chirp” (Google Cloud Speech-to-Text V2, 2023), which Google describes as its 2B-parameter USM

Google publishes no separate Chirp paper: the deployed model is USM. This class is therefore USM-CTC: the paper’s Conformer-2B (Table 2: 32 layers, 1536 wide, 16 heads, kernel 5) over 128-dimensional log-mel features (section 2.7) with BEST-RQ’s 4x convolutional reduction, 8-second chunk-wise attention (section 2.4) and a CTC layer. Pre-training (Pretrain) is BEST-RQ with the 16-codebook multi-softmax loss (sections 2.2.1 and 2.7). The paper’s Conformer uses relative attention; this one uses rotary position encoding, the relative scheme ConformerBlockLayer provides. The MOST speech-text stage and noisy student training of the USM paper are data procedures outside the network.

Example

// Create a Chirp recognizer
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.SpeechRecognition,
    inputHeight: 16000, inputWidth: 1, inputDepth: 1, outputSize: 32000);
var model = new Chirp<double>(architecture);
            
// Or load an exported ONNX model
var onnxModel = new Chirp<double>(architecture, "chirp.onnx");

Constructors

Constructor Summary
Chirp(NeuralNetworkArchitecture<T> architecture, ChirpOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
Chirp(NeuralNetworkArchitecture<T> architecture, string modelPath, ChirpOptions options)

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
T Pretrain(Tensor<T> audio) One BEST-RQ pre-training step (section 2.2) on waveforms [S] or [batch, S].
T PretrainingLoss(Tensor<T> audio, Random random) The BEST-RQ pre-training loss of waveforms [S] or [batch, S] with the multi-softmax of Zhang et al.
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes audio: the encoder, projection and CTC layer give per-frame distributions, decoded greedily.
string DetectLanguage(Tensor<T> audio)