CLAP<T>

Models & Types · AiDotNet.Audio.Classification

CLAP (Contrastive Language-Audio Pre-training) as an audio classifier: zero-shot against text prompts, or fine-tuned with a linear head.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → AudioClassifierBase<T> → CLAP<T>

Implements: IAudioEventDetector<T>

For Beginners

CLAP is special because it understands both audio and text. Instead of having fixed labels like “dog bark” or “siren”, you can describe any sound in plain English and CLAP will find it in audio. This is called “zero-shot” classification.

Usage:

How It Works

CLAP (Wu et al., ICASSP 2023) learns joint audio-text representations through contrastive learning, similar to CLIP for images. This class is the classification task over that network: it runs the audio and text towers of CLAPModel (built by the same factories from the same Network options), and adds

  • zero-shot classification: each prompt’s text embedding against the audio embedding, both

L2-normalized, scaled by 1/τ and turned into probabilities with a softmax over the prompts;

  • fine-tuning: LAION-CLAP’s linear probe (LinearProbe(mlp=False)), a linear layer on the

audio projection, trained with binary cross-entropy on logits, read through a sigmoid.

References:

  • Paper: “Large-Scale Contrastive Language-Audio Pre-Training with Feature Fusion and Keyword-to-Caption Augmentation” (Wu et al., ICASSP 2023)
  • Repository: https://github.com/LAION-AI/CLAP

Constructors

Constructor Summary
CLAP(NeuralNetworkArchitecture<T> architecture, CLAPOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a trainable CLAP classifier.
CLAP(NeuralNetworkArchitecture<T> architecture, string modelPath, CLAPOptions options) Creates a trainable CLAP classifier.

Properties

Property Summary
IReadOnlyList<string> EventLabels { get; }
IReadOnlyList<string> SupportedEvents { get; }
double TimeResolution { get; }

Methods

Method Summary
AudioEventResult<T> Detect(Tensor<T> audio)
AudioEventResult<T> Detect(Tensor<T> audio, T threshold)
AudioEventResult<T> DetectSpecific(Tensor<T> audio, IReadOnlyList<string> types)
AudioEventResult<T> DetectSpecific(Tensor<T> audio, IReadOnlyList<string> types, T threshold)
IStreamingEventDetectionSession<T> StartStreamingSession()
IStreamingEventDetectionSession<T> StartStreamingSession(int sr, T thr)
ModelMetadata<T> GetModelMetadata()
Task<AudioEventResult<T>> DetectAsync(Tensor<T> audio, CancellationToken ct)
Tensor<T> ExtractAudioEmbedding(Tensor<T> audio) Extracts the audio embedding in the joint audio-text space, L2-normalized.
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> GetEventProbabilities(Tensor<T> audio)
void SetTextPrompts(string[] prompts, Func<string, int[]> tokenize) Sets the text prompts for zero-shot classification.

Fields

Field Summary
string[] AudioSetLabels