SALMOptions

Options & Configuration · AiDotNet.SpeechRecognition.LLMIntegrated

Options for SALM: a Fast Conformer audio encoder, a Conformer modality adapter and a Megatron GPT decoder.

Inheritance: ModelOptions → SALMOptions

For Beginners

Defaults are the paper’s: the 110M Fast Conformer-large encoder (17 blocks of width 512, 8 heads, kernel 9, 8x depthwise-separable subsampling to 80 ms frames over 80 log-mel bands); two Conformer layers of width 512 with 4x subsampling as the modality adapter, so a speech token covers 320 ms; and the 2B Megatron GPT (24 layers of width 2,048 with 16 heads, SwiGLU 5,440, rotary embeddings on half of each head, LayerNorm, untied 256,000-row embeddings) with LoRA of rank 128 on its fused query-key-value projection.

Constructors

Constructor Summary
SALMOptions() Initializes a new instance with default values.
SALMOptions(SALMOptions other) Initializes a new instance by copying from another instance.

Properties

Property Summary
OnnxModelOptions OnnxOptions { get; set; } Gets or sets the ONNX Runtime settings used when the model is loaded from an ONNX file.
SpeechLlmTrainingScope TrainingScope { get; set; } What training updates: every parameter by default; the paper freezes the LLM and trains the encoder, the adapter and LoRA (EncoderAdapterAndLora).
double LlmNormEpsilon { get; set; } Megatron GPT’s LayerNorm epsilon.
double LlmRopeTheta { get; set; } Gets or sets the rotary positional-embedding frequency base for the language-model.
double LoraAlpha { get; set; } LoRA alpha; NeMo scales the update by alpha / rank with alpha equal to the rank.
double LoraDropout { get; set; } Gets or sets the dropout probability of the LoRA adapter.
double PartialRotaryFactor { get; set; } The share of each head the rotary embedding turns (rotary_percentage 0.5).
int AdapterDim { get; set; } The modality adapter’s width (512).
int ConvKernel { get; set; } The depthwise convolution kernel of the encoder and the adapter (9).
int EncoderDim { get; set; } The Fast Conformer encoder’s width (512).
int FeedForwardMultiplier { get; set; } The feed-forward expansion of the encoder and the adapter (4).
int LlmDim { get; set; } Megatron GPT’s width (2,048).
int LlmFeedForwardDim { get; set; } Megatron GPT’s SwiGLU width (5,440).
int LoraRank { get; set; } The LoRA rank on the fused query-key-value projection (128, §3.1).
int MaxAudioLengthSeconds { get; set; } Gets or sets the longest clip, in seconds, of the reference configuration.
int MaxTextLength { get; set; } The longest transcript generated, in tokens (the release’s tokens_to_generate, 128).
int NumAdapterHeads { get; set; } The modality adapter’s attention heads (8).
int NumAdapterLayers { get; set; } The modality adapter’s Conformer layers (2, §3.1).
int NumAttentionHeads { get; set; } The encoder’s attention heads (8).
int NumEncoderLayers { get; set; } The encoder’s Conformer blocks (17).
int NumLlmAttentionHeads { get; set; } Megatron GPT’s attention heads (16).
int NumLlmLayers { get; set; } Megatron GPT’s layers (24).
int NumMels { get; set; } Log-mel bands (80).
int SampleRate { get; set; } Gets or sets the sample rate, in Hz, that input audio is expected at.
int SubsamplingChannels { get; set; } Channels of the encoder’s and the adapter’s subsampling convolutions (256).
int VocabSize { get; set; } The vocabulary rows (256,000).
int WarmupSteps { get; set; } The learning-rate warm-up steps (2,000, §4).
string Instruction { get; set; } The instruction after the speech.
string Language { get; set; } Gets or sets the language code used for speech processing.
string ModelPath { get; set; } Gets or sets the path to the model weights.
string[] Vocabulary { get; set; } Optional vocabulary mapping token ids to text when the vocabulary is not Megatron’s.