SALMONN<T>
Models & Types · AiDotNet.Audio.Multimodal
SALMONN dual-encoder audio-language model for speech and audio understanding.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → SALMONN<T>
Implements: IAudioLanguageModel<T>
For Beginners
SALMONN has two “ears”: one for speech (Whisper) and one for general sounds (BEATs). This means it can understand what people say AND non-speech sounds. Ask it “What is the person saying?” and it transcribes speech. Ask “What sounds are in the background?” and it identifies environmental audio.
Usage:
How It Works
SALMONN (Tang et al., 2024, Tsinghua/ByteDance) uses dual audio encoders: a Whisper speech encoder and a BEATs audio encoder, connected to a Vicuna LLM through a window-level Q-Former adapter. This gives it strong capability for both speech and general audio understanding tasks.
Constructors
| Constructor | Summary |
|---|---|
SALMONN(NeuralNetworkArchitecture<T> architecture, SALMONNOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a SALMONN model in native training mode. |
SALMONN(NeuralNetworkArchitecture<T> architecture, string modelPath, SALMONNOptions options) |
Creates a SALMONN model in native training mode. |
Properties
| Property | Summary |
|---|---|
double MaxAudioDurationSeconds { get; } |
|
int MaxResponseTokens { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<string> GetCapabilities() |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<string> UnderstandAsync(Tensor<T> audio, string prompt, int maxTokens, double temperature, CancellationToken cancellationToken) |
|
Tensor<T> ExtractAudioEmbeddings(Tensor<T> audio) |
|
string Caption(Tensor<T> audio, int maxTokens) |
|
string Understand(Tensor<T> audio, string prompt, int maxTokens, double temperature) |