AudioSep<T>

Models & Types · AiDotNet.Audio.Classification

AudioSep - foundation model for open-vocabulary audio separation and sound event detection (Liu et al., ICML 2024).

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → AudioClassifierBase<T> → AudioSep<T>

Implements: IAudioEventDetector<T>

For Beginners

AudioSep is like having a smart audio assistant that understands natural language. You can ask it to find any sound:

  • “Find the dog barking” - detects when dogs bark in the audio
  • “Extract the piano” - separates piano from other instruments
  • “Isolate the speech” - pulls out speech from noisy background

This is powerful because you’re not limited to a fixed set of 527 categories - you can describe any sound in your own words, and AudioSep will find it.

Usage:

How It Works

AudioSep (Liu et al., ICML 2024) is a foundation model that uses natural language queries to detect and separate sounds. Unlike fixed-vocabulary SED models, AudioSep can detect any sound described in text using CLAP (Contrastive Language-Audio Pretraining) embeddings to condition a U-Net separation network. It achieves state-of-the-art results on both sound separation (MUSDB18, LibriMix) and sound event detection (AudioSet, ESC-50) benchmarks.

Architecture:

  • CLAP text encoder: Encodes the text query (e.g., “dog barking”) into a 512-dim

embedding that captures the semantic meaning of the target sound.

  • CLAP audio encoder: Encodes the input audio mixture into the same embedding space,

enabling audio-text matching.

  • Conditioned U-Net: A separation network conditioned on the CLAP text embedding.

The text embedding is injected via FiLM (Feature-wise Linear Modulation) at each layer, telling the network which sound to extract.

  • Classification head: For SED mode, applies sigmoid activation to produce per-class

probabilities from the audio-text similarity scores.

Constructors

Constructor Summary
AudioSep(NeuralNetworkArchitecture<T> architecture, AudioSepOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates an AudioSep model for native training mode.
AudioSep(NeuralNetworkArchitecture<T> architecture, string modelPath, AudioSepOptions options) Creates an AudioSep model for native training mode.

Properties

Property Summary
IReadOnlyList<string> EventLabels { get; }
IReadOnlyList<string> SupportedEvents { get; }
double TimeResolution { get; }

Methods

Method Summary
AudioEventResult<T> Detect(Tensor<T> audio)
AudioEventResult<T> Detect(Tensor<T> audio, T threshold)
AudioEventResult<T> DetectSpecific(Tensor<T> audio, IReadOnlyList<string> eventTypes)
AudioEventResult<T> DetectSpecific(Tensor<T> audio, IReadOnlyList<string> eventTypes, T threshold)
IStreamingEventDetectionSession<T> StartStreamingSession()
IStreamingEventDetectionSession<T> StartStreamingSession(int sampleRate, T threshold)
ModelMetadata<T> GetModelMetadata()
Task<AudioEventResult<T>> DetectAsync(Tensor<T> audio, CancellationToken cancellationToken)
Tensor<T> GetEventProbabilities(Tensor<T> audio)

Fields

Field Summary
string[] AudioSetLabels AudioSet-527 standard event labels.