AudioEventDetector<T>

Models & Types · AiDotNet.Audio.Classification

Audio event detection model for identifying sounds in audio (AudioSet-style).

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → AudioClassifierBase<T> → AudioEventDetector<T>

Implements: IAudioEventDetector<T>

For Beginners

Audio event detection answers “What sounds are in this audio?”:

  • Human sounds: speech, laughter, coughing, footsteps
  • Animal sounds: dog barking, bird singing, cat meowing
  • Music: instruments, genres, singing
  • Environmental: traffic, rain, wind, construction

Usage with ONNX:

Usage with training:

How It Works

Detects various audio events like speech, music, environmental sounds, and more. Based on AudioSet ontology with 527+ event classes organized hierarchically.

Architecture: Hershey et al. 2017’s best classifier, ResNet-50 adapted to audio (section 3.3.5): the stride of 2 is removed from the first 7x7 convolution and the final average pool covers the 6 x 4 activations of a 96 x 64 log-mel patch (960 ms of 25 ms windows every 10 ms, 64 mel bins, log after a small offset). Batch normalization follows every convolution and one sigmoid unit per label replaces the softmax, trained with cross-entropy and Adam (section 3.1). Audio is classified in non-overlapping 960 ms windows.

Constructors

Constructor Summary
AudioEventDetector(AudioEventDetectorOptions options) Creates an AudioEventDetector with legacy options only (native mode).
AudioEventDetector(NeuralNetworkArchitecture<T> architecture, AudioEventDetectorOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates an AudioEventDetector for native training mode with the default objective.
AudioEventDetector(NeuralNetworkArchitecture<T> architecture, AudioEventDetectorOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates an AudioEventDetector for native training mode.
AudioEventDetector(NeuralNetworkArchitecture<T> architecture, string modelPath, AudioEventDetectorOptions options) Creates an AudioEventDetector for native training mode with the default objective.

Properties

Property Summary
IReadOnlyList<string> EventLabels { get; } Gets the event labels (alias for SupportedEvents for legacy API compatibility).
IReadOnlyList<string> SupportedEvents { get; } Gets the list of event types this model can detect.
double TimeResolution { get; } Gets the time resolution for event detection in seconds.

Methods

Method Summary
AudioEventResult<T> Detect(Tensor<T> audio) Detects audio events in the audio stream.
AudioEventResult<T> Detect(Tensor<T> audio, T threshold) Detects audio events with custom threshold.
AudioEventResult<T> DetectSpecific(Tensor<T> audio, IReadOnlyList<string> eventTypes) Detects specific events only.
AudioEventResult<T> DetectSpecific(Tensor<T> audio, IReadOnlyList<string> eventTypes, T threshold) Detects specific events only with custom threshold.
Dictionary<string, double> DetectFrame(Tensor<T> audio) Detects a single frame (no windowing) - legacy API.
IStreamingEventDetectionSession<T> StartStreamingSession() Starts a streaming event detection session.
IStreamingEventDetectionSession<T> StartStreamingSession(int sampleRate, T threshold) Starts a streaming event detection session with custom settings.
List<AudioEvent> DetectLegacy(Tensor<T> audio) Detects audio events in the given audio (legacy API).
List<ValueTuple<string, double>> DetectTopK(Tensor<T> audio, int topK) Gets the top K events for a single frame (legacy API).
ModelMetadata<T> GetModelMetadata() Gets model metadata.
Task<AudioEventResult<T>> DetectAsync(Tensor<T> audio, CancellationToken cancellationToken) Detects audio events asynchronously.
Task<List<AudioEvent>> DetectLegacyAsync(Tensor<T> audio, CancellationToken cancellationToken) Detects audio events asynchronously (legacy API).
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> GetEventProbabilities(Tensor<T> audio) Gets frame-level event probabilities.

Fields

Field Summary
string[] CommonEventLabels Common audio event categories from AudioSet.