SALMONNOptions
Options & Configuration · AiDotNet.Audio.Multimodal
Configuration options for the SALMONN multimodal audio-language model.
Inheritance: ModelOptions → SALMONNOptions
For Beginners
SALMONN has two “ears”: one specialized for speech (Whisper) and one for general sounds (BEATs). This means it can understand both what people say AND non-speech sounds like music, animal sounds, and environmental noise. It’s like having a translator who also happens to be an expert sound engineer.
How It Works
SALMONN (Tang et al., 2024, Tsinghua/ByteDance) is a large language model with dual audio encoders: a Whisper speech encoder and a BEATs audio encoder, connected to a Vicuna LLM through a window-level Q-Former adapter. This dual-encoder design gives it strong capability for both speech understanding and general audio understanding tasks.
Constructors
| Constructor | Summary |
|---|---|
SALMONNOptions() |
Properties
| Property | Summary |
|---|---|
OnnxModelOptions OnnxOptions { get; set; } |
Gets or sets the ONNX runtime options. |
double DropoutRate { get; set; } |
Gets or sets the dropout rate. |
double LearningRate { get; set; } |
Gets or sets the learning rate. |
double MaxAudioDurationSeconds { get; set; } |
Gets or sets the maximum audio duration in seconds. |
double Temperature { get; set; } |
Gets or sets the sampling temperature. |
double TopP { get; set; } |
Gets or sets the top-p (nucleus) sampling parameter. |
int AudioEncoderDim { get; set; } |
Gets or sets the audio encoder dimension (BEATs). |
int LMHiddenDim { get; set; } |
Gets or sets the language model hidden dimension. |
int MaxResponseTokens { get; set; } |
Gets or sets the maximum response length in tokens. |
int NumAudioEncoderLayers { get; set; } |
Gets or sets the number of audio encoder layers. |
int NumLMHeads { get; set; } |
Gets or sets the number of language model attention heads. |
int NumLMLayers { get; set; } |
Gets or sets the number of language model layers. |
int NumMels { get; set; } |
Gets or sets the number of mel spectrogram channels. |
int NumQFormerLayers { get; set; } |
Gets or sets the number of Q-Former layers. |
int NumQueryTokens { get; set; } |
Gets or sets the number of Q-Former query tokens. |
int NumSpeechEncoderLayers { get; set; } |
Gets or sets the number of speech encoder layers. |
int QFormerDim { get; set; } |
Gets or sets the Q-Former hidden dimension. |
int SampleRate { get; set; } |
Gets or sets the audio sample rate in Hz. |
int SpeechEncoderDim { get; set; } |
Gets or sets the speech encoder dimension (Whisper). |
int VocabSize { get; set; } |
Gets or sets the LM vocabulary size. |
int WindowSize { get; set; } |
Gets or sets the window size for window-level Q-Former. |
string ModelPath { get; set; } |
Gets or sets the path to the ONNX model file. |