VoxtLMOptions

Options & Configuration · AiDotNet.SpeechRecognition.Specialized

Options for VoxtLM: one decoder-only language model over speech units and text.

Inheritance: ModelOptions → VoxtLMOptions

For Beginners

Defaults are the paper’s VoxtLM-k50 small model (Maiti et al. 2024, §4, the 125M rows of Table 3) as ESPnet’s egs2/voxtlm_v1 recipe trains it: speech becomes one of 50 k-means clusters of HuBERT Base’s sixth-layer features every 20 ms; text is one of 2,000 subword tokens; and a 12-layer Transformer language model of width 768 (12 heads, feed-forward 3,072, no positional encoding, dropout 0.1) predicts both.

Constructors

Constructor Summary
VoxtLMOptions() Initializes a new instance with default values.
VoxtLMOptions(VoxtLMOptions other) Initializes a new instance by copying from another instance.

Properties

Property Summary
OnnxModelOptions OnnxOptions { get; set; } Gets or sets the ONNX Runtime settings used when the model is loaded from an ONNX file.
double DropoutRate { get; set; } Dropout on the input layer, the residual branches, the attention weights and the feed-forward block (0.1).
double LabelSmoothing { get; set; } Label smoothing of the loss (0.1, the recipe’s lsm_weight).
double LearningRate { get; set; } Adam’s peak learning rate (1e-3, the recipe’s), reached at the end of warm-up.
double MaxGradientNorm { get; set; } The gradient-norm clip (5).
double WeightDecay { get; set; } Adam’s L2 weight decay (1e-6).
int EmbeddingDim { get; set; } The token embedding’s width (768, the recipe’s embed_unit).
int FeedForwardDim { get; set; } The feed-forward width (3,072, the recipe’s unit).
int HubertConvChannels { get; set; } The channels of HuBERT’s convolutional feature encoder (512).
int HubertHeads { get; set; } HuBERT Base’s attention heads (12).
int HubertIntermediate { get; set; } HuBERT Base’s feed-forward width (3,072).
int HubertLayer { get; set; } The HuBERT layer clustered (6, the recipe’s kmeans_feature: HuBERT_base/6).
int HubertPositionalGroups { get; set; } The groups of HuBERT’s convolutional position embedding (16).
int HubertPositionalKernel { get; set; } The kernel of HuBERT’s convolutional position embedding (128).
int HubertWidth { get; set; } HuBERT Base’s width (768).
int MaxAudioLengthSeconds { get; set; } Gets or sets the longest clip, in seconds, of the reference configuration.
int MaxTextLength { get; set; } The longest transcript generated, in tokens (150, the recipe’s ASR maxlen).
int ModelDim { get; set; } The Transformer’s width (768, the small model’s F).
int NumAttentionHeads { get; set; } The attention heads (12, the small model’s H).
int NumLayers { get; set; } The Transformer layers (12, the small model’s L).
int NumSpeechUnits { get; set; } The speech tokenizer’s k-means clusters (50, VoxtLM-k50; the paper also trains 200 and 1,000).
int SampleRate { get; set; } Gets or sets the sample rate, in Hz, that input audio is expected at.
int TextVocabSize { get; set; } The text subword tokens (2,000, the paper’s BPE size for k = 50).
int WarmupSteps { get; set; } The warmuplr schedule’s warm-up steps (10,000).
string Language { get; set; } Gets or sets the language code used for speech processing.
string ModelPath { get; set; } Gets or sets the path to the model weights.
string[] Vocabulary { get; set; } Optional text for each text-token index; without it, index t decodes as code point t.