AudioLDMClassifier<T>

Models & Types · AiDotNet.Audio.Classification

AudioLDM Classifier that repurposes AudioLDM’s latent representations for audio event detection.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → AudioClassifierBase<T> → AudioLDMClassifier<T>

Implements: IAudioEventDetector<T>

For Beginners

AudioLDM was originally built to create audio from text descriptions. But it turns out the “understanding” it developed during training is also great for recognizing sounds. This classifier reuses that understanding: instead of generating audio, it identifies what sounds are present in a recording.

Usage:

How It Works

AudioLDM Classifier (Liu et al., 2023) extracts intermediate features from the AudioLDM diffusion U-Net to build a strong audio classifier. Since AudioLDM was trained to generate audio from text, its internal representations capture rich, semantically meaningful audio features that transfer well to classification tasks.

Constructors

Constructor Summary
AudioLDMClassifier(NeuralNetworkArchitecture<T> architecture, AudioLDMClassifierOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates an AudioLDM Classifier in native training mode.
AudioLDMClassifier(NeuralNetworkArchitecture<T> architecture, string modelPath, AudioLDMClassifierOptions options) Creates an AudioLDM Classifier in native training mode.

Properties

Property Summary
IReadOnlyList<string> StandardLabels { get; } Standard AudioSet labels for audio event detection.
IReadOnlyList<string> SupportedEvents { get; }
bool IsOnnxMode { get; }
double TimeResolution { get; }

Methods

Method Summary
AudioEventResult<T> Detect(Tensor<T> audio)
AudioEventResult<T> Detect(Tensor<T> audio, T threshold)
AudioEventResult<T> DetectSpecific(Tensor<T> audio, IReadOnlyList<string> eventTypes)
AudioEventResult<T> DetectSpecific(Tensor<T> audio, IReadOnlyList<string> eventTypes, T threshold)
IStreamingEventDetectionSession<T> StartStreamingSession()
IStreamingEventDetectionSession<T> StartStreamingSession(int sampleRate, T threshold)
ModelMetadata<T> GetModelMetadata()
Task<AudioEventResult<T>> DetectAsync(Tensor<T> audio, CancellationToken cancellationToken)
Tensor<T> GetEventProbabilities(Tensor<T> audio)