CLAP<T>
Models & Types · AiDotNet.Audio.Classification
CLAP (Contrastive Language-Audio Pre-training) as an audio classifier: zero-shot against text prompts, or fine-tuned with a linear head.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → AudioClassifierBase<T> → CLAP<T>
Implements: IAudioEventDetector<T>
For Beginners
CLAP is special because it understands both audio and text. Instead of having fixed labels like “dog bark” or “siren”, you can describe any sound in plain English and CLAP will find it in audio. This is called “zero-shot” classification.
Usage:
How It Works
CLAP (Wu et al., ICASSP 2023) learns joint audio-text representations through contrastive learning,
similar to CLIP for images. This class is the classification task over that network: it runs the audio and
text towers of CLAPModel (built by the same factories from the same
Network options), and adds
- zero-shot classification: each prompt’s text embedding against the audio embedding, both
L2-normalized, scaled by 1/τ and turned into probabilities with a softmax over the prompts;
- fine-tuning: LAION-CLAP’s linear probe (
LinearProbe(mlp=False)), a linear layer on the
audio projection, trained with binary cross-entropy on logits, read through a sigmoid.
References:
- Paper: “Large-Scale Contrastive Language-Audio Pre-Training with Feature Fusion and Keyword-to-Caption Augmentation” (Wu et al., ICASSP 2023)
- Repository:
https://github.com/LAION-AI/CLAP
Constructors
| Constructor | Summary |
|---|---|
CLAP(NeuralNetworkArchitecture<T> architecture, CLAPOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a trainable CLAP classifier. |
CLAP(NeuralNetworkArchitecture<T> architecture, string modelPath, CLAPOptions options) |
Creates a trainable CLAP classifier. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> EventLabels { get; } |
|
IReadOnlyList<string> SupportedEvents { get; } |
|
double TimeResolution { get; } |
Methods
| Method | Summary |
|---|---|
AudioEventResult<T> Detect(Tensor<T> audio) |
|
AudioEventResult<T> Detect(Tensor<T> audio, T threshold) |
|
AudioEventResult<T> DetectSpecific(Tensor<T> audio, IReadOnlyList<string> types) |
|
AudioEventResult<T> DetectSpecific(Tensor<T> audio, IReadOnlyList<string> types, T threshold) |
|
IStreamingEventDetectionSession<T> StartStreamingSession() |
|
IStreamingEventDetectionSession<T> StartStreamingSession(int sr, T thr) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<AudioEventResult<T>> DetectAsync(Tensor<T> audio, CancellationToken ct) |
|
Tensor<T> ExtractAudioEmbedding(Tensor<T> audio) |
Extracts the audio embedding in the joint audio-text space, L2-normalized. |
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Tensor<T> GetEventProbabilities(Tensor<T> audio) |
|
void SetTextPrompts(string[] prompts, Func<string, int[]> tokenize) |
Sets the text prompts for zero-shot classification. |
Fields
| Field | Summary |
|---|---|
string[] AudioSetLabels |