GoogleSpeechV2<T>
Models & Types · AiDotNet.SpeechRecognition.ProprietaryAPI
Google Cloud Speech-to-Text V2: Google’s hosted recognizer, served on its Chirp models, which are the Universal Speech Model (USM) with a CTC layer.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → GoogleSpeechV2<T>
Implements: ISpeechRecognizer<T>
For Beginners
Speech-to-Text V2 is Google’s cloud transcription service. It listens to 16 kHz audio turned into a 128-band spectrogram, encodes it with a large Conformer, and outputs for every 40 ms frame a probability for each token; collapsing repeats and blanks gives the transcript.
How It Works
References:
- Paper: “Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages” (Zhang et al., Google, 2023)
- Service: Google Cloud Speech-to-Text V2, whose Chirp models Google describes as its 2B-parameter USM
The service’s published architecture is its Chirp model, USM. This class is therefore USM-CTC: the paper’s
Conformer-2B (Table 2: 32 layers, 1536 wide, 16 heads, kernel 5) over 128-dimensional log-mel features (section 2.7)
with BEST-RQ’s 4x convolutional reduction, 8-second chunk-wise attention (section 2.4) and a CTC layer, with BEST-RQ
pre-training (Pretrain) under the 16-codebook multi-softmax loss. Rotary position encoding stands in
for the paper’s relative attention.
The MOST speech-text stage and noisy student training of the USM paper are data procedures outside the network.
Example
// Create a Speech-to-Text V2 recognizer
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.SpeechRecognition,
inputHeight: 16000, inputWidth: 1, inputDepth: 1, outputSize: 32000);
var model = new GoogleSpeechV2<double>(architecture);
// Or load an exported ONNX model
var onnxModel = new GoogleSpeechV2<double>(architecture, "google-speech-v2.onnx");
Constructors
| Constructor | Summary |
|---|---|
GoogleSpeechV2(NeuralNetworkArchitecture<T> architecture, GoogleSpeechV2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
GoogleSpeechV2(NeuralNetworkArchitecture<T> architecture, string modelPath, GoogleSpeechV2Options options) |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
|
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
ModelMetadata<T> GetModelMetadata() |
|
T Pretrain(Tensor<T> audio) |
One BEST-RQ pre-training step (section 2.2) on waveforms [S] or [batch, S]. |
T PretrainingLoss(Tensor<T> audio, Random random) |
The BEST-RQ pre-training loss of waveforms [S] or [batch, S] with the multi-softmax of Zhang et al. |
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
Transcribes audio: the encoder, projection and CTC layer give per-frame distributions, decoded greedily. |
string DetectLanguage(Tensor<T> audio) |