SALMONN<T>

Models & Types · AiDotNet.Audio.Multimodal

SALMONN dual-encoder audio-language model for speech and audio understanding.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → SALMONN<T>

Implements: IAudioLanguageModel<T>

For Beginners

SALMONN has two “ears”: one for speech (Whisper) and one for general sounds (BEATs). This means it can understand what people say AND non-speech sounds. Ask it “What is the person saying?” and it transcribes speech. Ask “What sounds are in the background?” and it identifies environmental audio.

Usage:

How It Works

SALMONN (Tang et al., 2024, Tsinghua/ByteDance) uses dual audio encoders: a Whisper speech encoder and a BEATs audio encoder, connected to a Vicuna LLM through a window-level Q-Former adapter. This gives it strong capability for both speech and general audio understanding tasks.

Constructors

Constructor Summary
SALMONN(NeuralNetworkArchitecture<T> architecture, SALMONNOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a SALMONN model in native training mode.
SALMONN(NeuralNetworkArchitecture<T> architecture, string modelPath, SALMONNOptions options) Creates a SALMONN model in native training mode.

Properties

Property Summary
double MaxAudioDurationSeconds { get; }
int MaxResponseTokens { get; }

Methods

Method Summary
IReadOnlyList<string> GetCapabilities()
ModelMetadata<T> GetModelMetadata()
Task<string> UnderstandAsync(Tensor<T> audio, string prompt, int maxTokens, double temperature, CancellationToken cancellationToken)
Tensor<T> ExtractAudioEmbeddings(Tensor<T> audio)
string Caption(Tensor<T> audio, int maxTokens)
string Understand(Tensor<T> audio, string prompt, int maxTokens, double temperature)