AudioPaLM<T>
Models & Types · AiDotNet.SpeechRecognition.LLMIntegrated
AudioPaLM’s speech recognition (Rubenstein et al. 2023): the same decoder over text and audio tokens as
AudioPaLM, given [ASR language] and a recording’s audio tokens,
writing the transcript.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → AudioPaLM<T>
Implements: ISpeechRecognizer<T>
For Beginners
AudioPaLM is a language model that also understands “audio words”. To transcribe, it turns speech into audio words and writes the text that follows the instruction “[ASR English]”.
How It Works
The recording becomes 25 Hz audio tokens (w2v-BERT’s k-means codes, §3.1), offset past the text vocabulary (§3.2), after the task tag in the text tokenizer’s tokens (§3.4); the decoder continues with the transcript’s text tokens and an end token. Training is AudioPaLM’s: the cross-entropy of the transcript with the input’s positions masked, Adafactor at a constant 5e-5 (§3.6).
Example
var options = AudioPaLMOptions.Gemma2B();
var model = new AudioPaLM<float>(new NeuralNetworkArchitecture<float>(InputType.OneDimensional,
NeuralNetworkTaskType.SpeechRecognition, inputSize: 1, outputSize: 1), options);
model.LoadGemmaWeights("gemma-2b");
model.FitSpeechTokenizer(recordings);
model.Train(recording, transcriptIds);
var result = model.Transcribe(recording, "en");
Constructors
| Constructor | Summary |
|---|---|
AudioPaLM(NeuralNetworkArchitecture<T> architecture, AudioPaLMOptions options) |
Creates AudioPaLM’s speech recognizer. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
|
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
LayerInputDomain GetOutputDomain(int[] outputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
|
int[] TokenizeText(string text) |
The text token ids of text. |
string DetectLanguage(Tensor<T> audio) |
|
void FitSpeechTokenizer(IEnumerable<Tensor<T>> recordings) |
Fits the audio tokens’ k-means codebook on 16 kHz recordings (§3.1). |
void LoadGemmaWeights(string directory) |
Loads a Hugging Face Gemma checkpoint directory (use Gemma2B). |
void LoadPalmPytorchWeights(string path) |
Loads a lucidrains/PaLM-pytorch state dict. |
void LoadSpeechTokenizerCodebook(double[] centroids) |
Sets the w2v-BERT 2.0 tokenizer’s k-means centroids. |
void LoadW2vBert2Weights(string path) |
Loads w2v-BERT 2.0 for the w2v-BERT 2.0 tokenizer. |