CLAPModel<T>

Models & Types · AiDotNet.Audio.Fingerprinting

CLAP (Contrastive Language-Audio Pretraining) — a dual-encoder neural network that learns to align audio and text representations in a shared embedding space via a contrastive objective.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → CLAPModel<T>

Implements: IAudioFingerprinter<T>

How It Works

CLAP (Wu et al. 2023) trains an audio encoder and a text encoder so that matching (audio, caption) pairs produce nearby embeddings and unrelated pairs produce distant embeddings. The network is LAION-CLAP’s (reference LAION-AI/CLAP):

A learnable temperature τ scales the cosine-similarity logits during contrastive training (CLIP / CLAP convention). Published LAION checkpoints load through LoadLaionWeights.

Capabilities:

  • Zero-shot audio classification with text prompts
  • Audio-to-text and text-to-audio retrieval
  • Semantic audio fingerprinting via the projection head

Reference: Wu, Y. et al. (2023), “Large-Scale Contrastive Language-Audio Pre-Training with Feature Fusion and Keyword-to-Caption Augmentation”, ICASSP.

Constructors

Constructor Summary
CLAPModel(NeuralNetworkArchitecture<T> architecture, CLAPModelOptions options) Initializes a new instance of CLAPModel in native training / inference mode.
CLAPModel(NeuralNetworkArchitecture<T> architecture, string audioEncoderPath, CLAPModelOptions options) Initializes a new instance of CLAPModel in ONNX inference mode.

Properties

Property Summary
double LogitScale { get; } The learned logit scale exp(logit_scale_a) = 1 / τ.
int FingerprintLength { get; }
string Name { get; }

Methods

Method Summary
AudioFingerprint<T> Fingerprint(Tensor<T> audio)
AudioFingerprint<T> Fingerprint(Vector<T> audio)
Dictionary<string, double> ZeroShotClassify(Tensor<T> audio, string[] labels, Func<string, int[]> tokenize) Performs zero-shot audio classification: rank each text prompt by its CLAP-similarity to the supplied audio clip.
Dictionary<string, double> ZeroShotClassify(Tensor<T> audio, string[] labels, ITokenizer tokenizer) Performs zero-shot audio classification: rank each text prompt by its CLAP-similarity to the supplied audio clip.
IReadOnlyCollection<string> LoadLaionWeights(Func<string, int[], double[]> read) Loads a LAION-CLAP checkpoint straight from source; returns the names read.
IReadOnlyList<FingerprintMatch> FindMatches(AudioFingerprint<T> query, AudioFingerprint<T> reference, int minMatchLength)
ModelMetadata<T> GetModelMetadata()
Tensor<T> ComputeLogMel(Tensor<T> waveforms) The log-mel spectrogram of each row of waveforms, [batch, frames, mels], computed with the filterbank of this model variant and no clip fitting.
Tensor<T> EncodeAudio(Tensor<T> audio) Encodes audio into a CLAP embedding vector in the shared text-audio space.
Tensor<T> EncodeLogMel(Tensor<T> mel, Tensor<T> longer) Encodes log-mel input directly, bypassing clip fitting: mel is [batch, views, frames, mels] (one view, or the four fusion views) and longer the optional [batch] 0/1 mask of samples whose local views are fused.
Tensor<T> EncodeText(IReadOnlyList<string> captions, ITokenizer tokenizer) Tokenizes captions as LAION-CLAP does — special tokens added, truncated and padded to MaxTextLength — and encodes them.
Tensor<T> EncodeText(Tensor<T> tokens) Convenience overload: encodes one caption’s token ids.
Tensor<T> EncodeText(int[] tokenIds) Convenience overload: encodes one caption’s token ids.
Tensor<T> ForwardForTraining(Tensor<T> input) The audio embedding in training mode: the forward that TryApplyLearningRule trains.
Tensor<T> Tokenize(IReadOnlyList<string> captions, ITokenizer tokenizer) Token ids [captions, MaxTextLength] for captions: tokenizer(text, padding="max_length", truncation=True, max_length=77) in LAION-CLAP’s hook.py.
double ComputeSimilarity(AudioFingerprint<T> fp1, AudioFingerprint<T> fp2)