VoxtLMOptions
Options & Configuration · AiDotNet.SpeechRecognition.Specialized
Options for VoxtLM: one decoder-only language model over speech units and text.
Inheritance: ModelOptions → VoxtLMOptions
For Beginners
Defaults are the paper’s VoxtLM-k50 small model (Maiti et al. 2024, §4, the 125M rows
of Table 3) as ESPnet’s egs2/voxtlm_v1 recipe trains it: speech becomes one of 50 k-means clusters of
HuBERT Base’s sixth-layer features every 20 ms; text is one of 2,000 subword tokens; and a 12-layer Transformer
language model of width 768 (12 heads, feed-forward 3,072, no positional encoding, dropout 0.1) predicts both.
Constructors
| Constructor | Summary |
|---|---|
VoxtLMOptions() |
Initializes a new instance with default values. |
VoxtLMOptions(VoxtLMOptions other) |
Initializes a new instance by copying from another instance. |
Properties
| Property | Summary |
|---|---|
OnnxModelOptions OnnxOptions { get; set; } |
Gets or sets the ONNX Runtime settings used when the model is loaded from an ONNX file. |
double DropoutRate { get; set; } |
Dropout on the input layer, the residual branches, the attention weights and the feed-forward block (0.1). |
double LabelSmoothing { get; set; } |
Label smoothing of the loss (0.1, the recipe’s lsm_weight). |
double LearningRate { get; set; } |
Adam’s peak learning rate (1e-3, the recipe’s), reached at the end of warm-up. |
double MaxGradientNorm { get; set; } |
The gradient-norm clip (5). |
double WeightDecay { get; set; } |
Adam’s L2 weight decay (1e-6). |
int EmbeddingDim { get; set; } |
The token embedding’s width (768, the recipe’s embed_unit). |
int FeedForwardDim { get; set; } |
The feed-forward width (3,072, the recipe’s unit). |
int HubertConvChannels { get; set; } |
The channels of HuBERT’s convolutional feature encoder (512). |
int HubertHeads { get; set; } |
HuBERT Base’s attention heads (12). |
int HubertIntermediate { get; set; } |
HuBERT Base’s feed-forward width (3,072). |
int HubertLayer { get; set; } |
The HuBERT layer clustered (6, the recipe’s kmeans_feature: HuBERT_base/6). |
int HubertPositionalGroups { get; set; } |
The groups of HuBERT’s convolutional position embedding (16). |
int HubertPositionalKernel { get; set; } |
The kernel of HuBERT’s convolutional position embedding (128). |
int HubertWidth { get; set; } |
HuBERT Base’s width (768). |
int MaxAudioLengthSeconds { get; set; } |
Gets or sets the longest clip, in seconds, of the reference configuration. |
int MaxTextLength { get; set; } |
The longest transcript generated, in tokens (150, the recipe’s ASR maxlen). |
int ModelDim { get; set; } |
The Transformer’s width (768, the small model’s F). |
int NumAttentionHeads { get; set; } |
The attention heads (12, the small model’s H). |
int NumLayers { get; set; } |
The Transformer layers (12, the small model’s L). |
int NumSpeechUnits { get; set; } |
The speech tokenizer’s k-means clusters (50, VoxtLM-k50; the paper also trains 200 and 1,000). |
int SampleRate { get; set; } |
Gets or sets the sample rate, in Hz, that input audio is expected at. |
int TextVocabSize { get; set; } |
The text subword tokens (2,000, the paper’s BPE size for k = 50). |
int WarmupSteps { get; set; } |
The warmuplr schedule’s warm-up steps (10,000). |
string Language { get; set; } |
Gets or sets the language code used for speech processing. |
string ModelPath { get; set; } |
Gets or sets the path to the model weights. |
string[] Vocabulary { get; set; } |
Optional text for each text-token index; without it, index t decodes as code point t. |