VoxtLM<T>
Models & Types · AiDotNet.SpeechRecognition.Specialized
VoxtLM: one decoder-only language model for speech recognition, synthesis, and speech and text continuation (Maiti et al. 2024), here as a speech recognizer.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → VoxtLM<T>
Implements: ISpeechRecognizer<T>
For Beginners
VoxtLM treats speech as a second language. A speech tokenizer turns the recording into a
sequence of discrete “speech words” (clusters of HuBERT features), and one language model, trained on speech, text
and both together, reads <startspeech>, the speech words and <generatetext>, then writes
the transcript.
How It Works
The defaults are VoxtLM-k50 small as ESPnet’s egs2/voxtlm_v1 recipe trains it (§3–4 and the recipe’s
train.yaml): the speech tokenizer is k-means with 50 clusters over HuBERT Base’s sixth-layer features
(kmeans_feature: hubert_base/6), the text has 2,000 subword tokens, and the language model is ESPnet’s
TransformerLM of 12 pre-norm layers at width 768 (12 heads, ReLU feed-forward 3,072, no positional encoding,
dropout 0.1) with an untied output layer. Training predicts the transcript and <sos/eos> after the
condition (Table 1), with label smoothing 0.1, under Adam (1e-3, weight decay 1e-6) with the warmuplr schedule
over 10,000 steps and gradient clipping at 5. The speech tokenizer does not train: load HuBERT and a fitted k-means
(LoadSpeechTokenizer) or fit the k-means here (FitSpeechUnits). The paper decodes with
beam search (beam 5 in the recipe); Transcribe decodes greedily. The recipe also applies BPE over the
unit sequences; here each unit is one token.
Example
var architecture = new NeuralNetworkArchitecture<float>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.SpeechRecognition,
inputSize: 16000, outputSize: 2057);
var model = new VoxtLM<float>(architecture);
model.LoadSpeechTokenizer("hubert_base_ls960.pt", "km_50.mdl");
var text = model.Transcribe(waveform).Text;
Constructors
| Constructor | Summary |
|---|---|
VoxtLM(NeuralNetworkArchitecture<T> architecture, VoxtLMOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
VoxtLM(NeuralNetworkArchitecture<T> architecture, string modelPath, VoxtLMOptions options) |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
|
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
Transcribes audio: the language model’s greedy transcript after the ASR condition. |
string DetectLanguage(Tensor<T> audio) |
|
void FitSpeechUnits(IEnumerable<Tensor<T>> recordings, int iterations) |
Fits the speech tokenizer’s k-means on the HuBERT features of 16 kHz recordings. |
void LoadSpeechTokenizer(string hubertCheckpoint, string kmeansModel) |
Loads the speech tokenizer: a fairseq HuBERT checkpoint (HuBERT Base for the recipe’s hubert_base/6) and a joblib-saved scikit-learn k-means model fitted on its features. |