SALMOptions
Options & Configuration · AiDotNet.SpeechRecognition.LLMIntegrated
Options for SALM: a Fast Conformer audio encoder, a Conformer modality adapter and a Megatron GPT decoder.
Inheritance: ModelOptions → SALMOptions
For Beginners
Defaults are the paper’s: the 110M Fast Conformer-large encoder (17 blocks of width 512, 8 heads, kernel 9, 8x depthwise-separable subsampling to 80 ms frames over 80 log-mel bands); two Conformer layers of width 512 with 4x subsampling as the modality adapter, so a speech token covers 320 ms; and the 2B Megatron GPT (24 layers of width 2,048 with 16 heads, SwiGLU 5,440, rotary embeddings on half of each head, LayerNorm, untied 256,000-row embeddings) with LoRA of rank 128 on its fused query-key-value projection.
Constructors
| Constructor | Summary |
|---|---|
SALMOptions() |
Initializes a new instance with default values. |
SALMOptions(SALMOptions other) |
Initializes a new instance by copying from another instance. |
Properties
| Property | Summary |
|---|---|
OnnxModelOptions OnnxOptions { get; set; } |
Gets or sets the ONNX Runtime settings used when the model is loaded from an ONNX file. |
SpeechLlmTrainingScope TrainingScope { get; set; } |
What training updates: every parameter by default; the paper freezes the LLM and trains the encoder, the adapter and LoRA (EncoderAdapterAndLora). |
double LlmNormEpsilon { get; set; } |
Megatron GPT’s LayerNorm epsilon. |
double LlmRopeTheta { get; set; } |
Gets or sets the rotary positional-embedding frequency base for the language-model. |
double LoraAlpha { get; set; } |
LoRA alpha; NeMo scales the update by alpha / rank with alpha equal to the rank. |
double LoraDropout { get; set; } |
Gets or sets the dropout probability of the LoRA adapter. |
double PartialRotaryFactor { get; set; } |
The share of each head the rotary embedding turns (rotary_percentage 0.5). |
int AdapterDim { get; set; } |
The modality adapter’s width (512). |
int ConvKernel { get; set; } |
The depthwise convolution kernel of the encoder and the adapter (9). |
int EncoderDim { get; set; } |
The Fast Conformer encoder’s width (512). |
int FeedForwardMultiplier { get; set; } |
The feed-forward expansion of the encoder and the adapter (4). |
int LlmDim { get; set; } |
Megatron GPT’s width (2,048). |
int LlmFeedForwardDim { get; set; } |
Megatron GPT’s SwiGLU width (5,440). |
int LoraRank { get; set; } |
The LoRA rank on the fused query-key-value projection (128, §3.1). |
int MaxAudioLengthSeconds { get; set; } |
Gets or sets the longest clip, in seconds, of the reference configuration. |
int MaxTextLength { get; set; } |
The longest transcript generated, in tokens (the release’s tokens_to_generate, 128). |
int NumAdapterHeads { get; set; } |
The modality adapter’s attention heads (8). |
int NumAdapterLayers { get; set; } |
The modality adapter’s Conformer layers (2, §3.1). |
int NumAttentionHeads { get; set; } |
The encoder’s attention heads (8). |
int NumEncoderLayers { get; set; } |
The encoder’s Conformer blocks (17). |
int NumLlmAttentionHeads { get; set; } |
Megatron GPT’s attention heads (16). |
int NumLlmLayers { get; set; } |
Megatron GPT’s layers (24). |
int NumMels { get; set; } |
Log-mel bands (80). |
int SampleRate { get; set; } |
Gets or sets the sample rate, in Hz, that input audio is expected at. |
int SubsamplingChannels { get; set; } |
Channels of the encoder’s and the adapter’s subsampling convolutions (256). |
int VocabSize { get; set; } |
The vocabulary rows (256,000). |
int WarmupSteps { get; set; } |
The learning-rate warm-up steps (2,000, §4). |
string Instruction { get; set; } |
The instruction after the speech. |
string Language { get; set; } |
Gets or sets the language code used for speech processing. |
string ModelPath { get; set; } |
Gets or sets the path to the model weights. |
string[] Vocabulary { get; set; } |
Optional vocabulary mapping token ids to text when the vocabulary is not Megatron’s. |