CLAPOptions
Options & Configuration · AiDotNet.Audio.Classification
Configuration options for the CLAP (Contrastive Language-Audio Pre-training) model.
Inheritance: ModelOptions → CLAPOptions
For Beginners
CLAP is special because it understands both audio and text. Instead of having fixed labels like “dog bark” or “siren”, you can describe any sound in plain English and CLAP will find it in audio. For example, you can search for “the sound of rain hitting a tin roof” without ever training on that specific label.
How it works:
- CLAP has two encoders: one for audio and one for text
- Both encoders map their inputs to the same shared space
- Matching audio-text pairs are placed close together in this space
- To classify audio, compare it with text descriptions and find the closest match
How It Works
CLAP (Wu et al., ICASSP 2023) learns joint audio-text representations through contrastive learning, similar to CLIP for images. It can classify audio using natural language descriptions without task-specific training (zero-shot classification), achieving 26.7% zero-shot accuracy on ESC-50 and 46.8% mAP on AudioSet with fine-tuning.
References:
- Paper: “Large-Scale Contrastive Language-Audio Pre-Training with Feature Fusion and Keyword-to-Caption Augmentation” (Wu et al., ICASSP 2023)
- Repository:
https://github.com/LAION-AI/CLAP
Constructors
| Constructor | Summary |
|---|---|
CLAPOptions() |
Properties
| Property | Summary |
|---|---|
CLAPModelOptions Network { get; set; } |
Gets or sets the CLAP network: preprocessing, the audio and text towers and the joint space. |
OnnxModelOptions OnnxOptions { get; set; } |
Gets or sets ONNX runtime options. |
double LearningRate { get; set; } |
Gets or sets the initial learning rate (Wu et al. |
double Threshold { get; set; } |
Gets or sets the confidence threshold. |
double WindowOverlap { get; set; } |
Gets or sets the window overlap ratio. |
double WindowSize { get; set; } |
Gets or sets the window size in seconds. |
string ModelPath { get; set; } |
Gets or sets the path to a pre-trained ONNX audio encoder. |
string TextEncoderModelPath { get; set; } |
Gets or sets the path to the ONNX text encoder, which zero-shot prompts need in ONNX mode. |
string[] CustomLabels { get; set; } |
Gets or sets the labels of the fine-tuning head (AudioSet’s 527 when null). |