AudioSepOptions

Options & Configuration · AiDotNet.Audio.Classification

Configuration options for the AudioSep (Audio Separation with Natural Language Queries) model.

Inheritance: ModelOptions → AudioSepOptions

For Beginners

AudioSep is like having a smart audio assistant that understands natural language. Instead of being limited to a fixed set of 527 sound categories:

  • You can ask: “Find the sound of a baby crying” and it will detect/separate it
  • You can ask: “Extract the bird chirping” from a noisy recording
  • You can describe any sound in your own words, and AudioSep understands

This is possible because AudioSep combines two powerful ideas:

  1. CLAP: A model that understands the relationship between sounds and text descriptions
  2. Separation network: A U-Net that extracts the described sound from the mixture

AudioSep achieves state-of-the-art results on both sound separation and sound event detection benchmarks, making it one of the most versatile audio models available.

How It Works

AudioSep (Liu et al., ICML 2024) is a foundation model for open-vocabulary audio separation and sound event detection. Unlike traditional SED models that use fixed label sets, AudioSep can detect and separate any sound described by natural language. It uses CLAP (Contrastive Language-Audio Pretraining) embeddings to condition a separation network, enabling queries like “separate the dog barking from the traffic noise” or “detect the sound of glass breaking.”

Constructors

Constructor Summary
AudioSepOptions()

Properties

Property Summary
OnnxModelOptions OnnxOptions { get; set; } Gets or sets ONNX runtime options.
double DetectionWindowSize { get; set; } Gets or sets the window size in seconds for detection.
double DropoutRate { get; set; } Gets or sets the dropout rate.
double LearningRate { get; set; } Gets or sets the learning rate.
double Threshold { get; set; } Gets or sets the confidence threshold for event detection.
double WindowOverlap { get; set; } Gets or sets the window overlap ratio (0-1).
int CLAPEmbeddingDim { get; set; } Gets or sets the CLAP embedding dimension.
int FMax { get; set; } Gets or sets the maximum frequency for mel filterbank.
int FMin { get; set; } Gets or sets the minimum frequency for mel filterbank.
int FftSize { get; set; } Gets or sets the FFT window size.
int HopLength { get; set; } Gets or sets the hop length between FFT frames.
int NumHeads { get; set; } Gets or sets the number of attention heads in the separator.
int NumMels { get; set; } Gets or sets the number of mel filterbank channels.
int NumSeparationLayers { get; set; } Gets or sets the number of separation network layers.
int SampleRate { get; set; } Gets or sets the audio sample rate in Hz.
int SeparationDim { get; set; } Gets or sets the separation network hidden dimension.
int[] EncoderChannels { get; set; } Gets or sets the U-Net encoder channels.
string ModelPath { get; set; } Gets or sets the path to a pre-trained ONNX model file.
string Variant { get; set; } Gets or sets the model variant (“base”, “large”).
string[] CustomLabels { get; set; } Gets or sets custom event labels.