SALMONNOptions

Options & Configuration · AiDotNet.Audio.Multimodal

Configuration options for the SALMONN multimodal audio-language model.

Inheritance: ModelOptions → SALMONNOptions

For Beginners

SALMONN has two “ears”: one specialized for speech (Whisper) and one for general sounds (BEATs). This means it can understand both what people say AND non-speech sounds like music, animal sounds, and environmental noise. It’s like having a translator who also happens to be an expert sound engineer.

How It Works

SALMONN (Tang et al., 2024, Tsinghua/ByteDance) is a large language model with dual audio encoders: a Whisper speech encoder and a BEATs audio encoder, connected to a Vicuna LLM through a window-level Q-Former adapter. This dual-encoder design gives it strong capability for both speech understanding and general audio understanding tasks.

Constructors

Constructor Summary
SALMONNOptions()

Properties

Property Summary
OnnxModelOptions OnnxOptions { get; set; } Gets or sets the ONNX runtime options.
double DropoutRate { get; set; } Gets or sets the dropout rate.
double LearningRate { get; set; } Gets or sets the learning rate.
double MaxAudioDurationSeconds { get; set; } Gets or sets the maximum audio duration in seconds.
double Temperature { get; set; } Gets or sets the sampling temperature.
double TopP { get; set; } Gets or sets the top-p (nucleus) sampling parameter.
int AudioEncoderDim { get; set; } Gets or sets the audio encoder dimension (BEATs).
int LMHiddenDim { get; set; } Gets or sets the language model hidden dimension.
int MaxResponseTokens { get; set; } Gets or sets the maximum response length in tokens.
int NumAudioEncoderLayers { get; set; } Gets or sets the number of audio encoder layers.
int NumLMHeads { get; set; } Gets or sets the number of language model attention heads.
int NumLMLayers { get; set; } Gets or sets the number of language model layers.
int NumMels { get; set; } Gets or sets the number of mel spectrogram channels.
int NumQFormerLayers { get; set; } Gets or sets the number of Q-Former layers.
int NumQueryTokens { get; set; } Gets or sets the number of Q-Former query tokens.
int NumSpeechEncoderLayers { get; set; } Gets or sets the number of speech encoder layers.
int QFormerDim { get; set; } Gets or sets the Q-Former hidden dimension.
int SampleRate { get; set; } Gets or sets the audio sample rate in Hz.
int SpeechEncoderDim { get; set; } Gets or sets the speech encoder dimension (Whisper).
int VocabSize { get; set; } Gets or sets the LM vocabulary size.
int WindowSize { get; set; } Gets or sets the window size for window-level Q-Former.
string ModelPath { get; set; } Gets or sets the path to the ONNX model file.