VoxtLM<T>

Models & Types · AiDotNet.SpeechRecognition.Specialized

VoxtLM: one decoder-only language model for speech recognition, synthesis, and speech and text continuation (Maiti et al. 2024), here as a speech recognizer.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → VoxtLM<T>

Implements: ISpeechRecognizer<T>

For Beginners

VoxtLM treats speech as a second language. A speech tokenizer turns the recording into a sequence of discrete “speech words” (clusters of HuBERT features), and one language model, trained on speech, text and both together, reads <startspeech>, the speech words and <generatetext>, then writes the transcript.

How It Works

The defaults are VoxtLM-k50 small as ESPnet’s egs2/voxtlm_v1 recipe trains it (§3–4 and the recipe’s train.yaml): the speech tokenizer is k-means with 50 clusters over HuBERT Base’s sixth-layer features (kmeans_feature: hubert_base/6), the text has 2,000 subword tokens, and the language model is ESPnet’s TransformerLM of 12 pre-norm layers at width 768 (12 heads, ReLU feed-forward 3,072, no positional encoding, dropout 0.1) with an untied output layer. Training predicts the transcript and <sos/eos> after the condition (Table 1), with label smoothing 0.1, under Adam (1e-3, weight decay 1e-6) with the warmuplr schedule over 10,000 steps and gradient clipping at 5. The speech tokenizer does not train: load HuBERT and a fitted k-means (LoadSpeechTokenizer) or fit the k-means here (FitSpeechUnits). The paper decodes with beam search (beam 5 in the recipe); Transcribe decodes greedily. The recipe also applies BPE over the unit sequences; here each unit is one token.

Example

var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.SpeechRecognition,
    inputSize: 16000, outputSize: 2057);
var model = new VoxtLM<float>(architecture);
model.LoadSpeechTokenizer("hubert_base_ls960.pt", "km_50.mdl");
var text = model.Transcribe(waveform).Text;

Constructors

Constructor Summary
VoxtLM(NeuralNetworkArchitecture<T> architecture, VoxtLMOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
VoxtLM(NeuralNetworkArchitecture<T> architecture, string modelPath, VoxtLMOptions options)

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes audio: the language model’s greedy transcript after the ASR condition.
string DetectLanguage(Tensor<T> audio)
void FitSpeechUnits(IEnumerable<Tensor<T>> recordings, int iterations) Fits the speech tokenizer’s k-means on the HuBERT features of 16 kHz recordings.
void LoadSpeechTokenizer(string hubertCheckpoint, string kmeansModel) Loads the speech tokenizer: a fairseq HuBERT checkpoint (HuBERT Base for the recipe’s hubert_base/6) and a joblib-saved scikit-learn k-means model fitted on its features.