CLAPModel<T>
Models & Types · AiDotNet.Audio.Fingerprinting
CLAP (Contrastive Language-Audio Pretraining) — a dual-encoder neural network that learns to align audio and text representations in a shared embedding space via a contrastive objective.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → CLAPModel<T>
Implements: IAudioFingerprinter<T>
How It Works
CLAP (Wu et al. 2023) trains an audio encoder and a text encoder so that
matching (audio, caption) pairs produce nearby embeddings and unrelated
pairs produce distant embeddings. The network is LAION-CLAP’s (reference LAION-AI/CLAP):
A learnable temperature τ scales the cosine-similarity logits during contrastive training (CLIP / CLAP
convention). Published LAION checkpoints load through LoadLaionWeights.
Capabilities:
- Zero-shot audio classification with text prompts
- Audio-to-text and text-to-audio retrieval
- Semantic audio fingerprinting via the projection head
Reference: Wu, Y. et al. (2023), “Large-Scale Contrastive Language-Audio Pre-Training with Feature Fusion and Keyword-to-Caption Augmentation”, ICASSP.
Constructors
| Constructor | Summary |
|---|---|
CLAPModel(NeuralNetworkArchitecture<T> architecture, CLAPModelOptions options) |
Initializes a new instance of CLAPModel in native training / inference mode. |
CLAPModel(NeuralNetworkArchitecture<T> architecture, string audioEncoderPath, CLAPModelOptions options) |
Initializes a new instance of CLAPModel in ONNX inference mode. |
Properties
| Property | Summary |
|---|---|
double LogitScale { get; } |
The learned logit scale exp(logit_scale_a) = 1 / τ. |
int FingerprintLength { get; } |
|
string Name { get; } |
Methods
| Method | Summary |
|---|---|
AudioFingerprint<T> Fingerprint(Tensor<T> audio) |
|
AudioFingerprint<T> Fingerprint(Vector<T> audio) |
|
Dictionary<string, double> ZeroShotClassify(Tensor<T> audio, string[] labels, Func<string, int[]> tokenize) |
Performs zero-shot audio classification: rank each text prompt by its CLAP-similarity to the supplied audio clip. |
Dictionary<string, double> ZeroShotClassify(Tensor<T> audio, string[] labels, ITokenizer tokenizer) |
Performs zero-shot audio classification: rank each text prompt by its CLAP-similarity to the supplied audio clip. |
IReadOnlyCollection<string> LoadLaionWeights(Func<string, int[], double[]> read) |
Loads a LAION-CLAP checkpoint straight from source; returns the names read. |
IReadOnlyList<FingerprintMatch> FindMatches(AudioFingerprint<T> query, AudioFingerprint<T> reference, int minMatchLength) |
|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> ComputeLogMel(Tensor<T> waveforms) |
The log-mel spectrogram of each row of waveforms, [batch, frames, mels], computed with the filterbank of this model variant and no clip fitting. |
Tensor<T> EncodeAudio(Tensor<T> audio) |
Encodes audio into a CLAP embedding vector in the shared text-audio space. |
Tensor<T> EncodeLogMel(Tensor<T> mel, Tensor<T> longer) |
Encodes log-mel input directly, bypassing clip fitting: mel is [batch, views, frames, mels] (one view, or the four fusion views) and longer the optional [batch] 0/1 mask of samples whose local views are fused. |
Tensor<T> EncodeText(IReadOnlyList<string> captions, ITokenizer tokenizer) |
Tokenizes captions as LAION-CLAP does — special tokens added, truncated and padded to MaxTextLength — and encodes them. |
Tensor<T> EncodeText(Tensor<T> tokens) |
Convenience overload: encodes one caption’s token ids. |
Tensor<T> EncodeText(int[] tokenIds) |
Convenience overload: encodes one caption’s token ids. |
Tensor<T> ForwardForTraining(Tensor<T> input) |
The audio embedding in training mode: the forward that TryApplyLearningRule trains. |
Tensor<T> Tokenize(IReadOnlyList<string> captions, ITokenizer tokenizer) |
Token ids [captions, MaxTextLength] for captions: tokenizer(text, padding="max_length", truncation=True, max_length=77) in LAION-CLAP’s hook.py. |
double ComputeSimilarity(AudioFingerprint<T> fp1, AudioFingerprint<T> fp2) |