IAudioEventDetector<T>

Interfaces · AiDotNet.Interfaces

Interface for audio event detection models that identify specific sounds/events in audio.

Implements: IFeatureAware, IFullModel<T, Tensor<T>, Tensor<T>>, IGradientComputable<T, Tensor<T>, Tensor<T>>

Known implementations: AST<T>, AudioEventDetector<T>, AudioLDMClassifier<T>, AudioMAE<T>, AudioSep<T>, BEATs<T>, CLAP<T>, CRNNEventDetector<T>, EAT<T>, FDYSED<T>, HTSAT<T>, PANNs<T>

For Beginners

Event detection is like having a listener who notes down every distinct sound they hear and when it happened.

How it works:

  1. Audio is analyzed in overlapping windows
  2. Each window is classified for the presence of various events
  3. Consecutive detections are merged into event segments

Types of events:

  • Environmental: Car horn, dog bark, siren, glass breaking
  • Speech: Laughter, cough, scream, applause
  • Music: Drum hit, guitar strum, piano note
  • Industrial: Machine alarm, tool sounds

Use cases:

  • Security/surveillance (detect gunshots, breaking glass)
  • Smart home (detect doorbell, smoke alarm, baby crying)
  • Wildlife monitoring (detect animal calls)
  • Content moderation (detect inappropriate sounds)
  • Accessibility (alert deaf users to sounds)

Challenges:

  • Overlapping events (multiple sounds at once)
  • Variable event duration (short beep vs long siren)
  • Background noise interference

How It Works

Audio event detection identifies when specific sounds occur in an audio stream. Unlike classification which assigns one label to entire clips, event detection finds multiple events with their timestamps.

This interface extends IFullModel for Tensor-based audio processing.

Properties

Property Summary
IReadOnlyList<string> SupportedEvents { get; } Gets the list of event types this model can detect.
bool IsOnnxMode { get; } Gets whether this model is running in ONNX inference mode.
double TimeResolution { get; } Gets the time resolution for event detection in seconds.
int SampleRate { get; } Gets the expected sample rate for input audio.

Methods

Method Summary
AudioEventResult<T> Detect(Tensor<T> audio) Detects audio events in the audio stream.
AudioEventResult<T> Detect(Tensor<T> audio, T threshold) Detects audio events in the audio stream with custom threshold.
AudioEventResult<T> DetectSpecific(Tensor<T> audio, IReadOnlyList<string> eventTypes) Detects specific events only.
AudioEventResult<T> DetectSpecific(Tensor<T> audio, IReadOnlyList<string> eventTypes, T threshold) Detects specific events only with custom threshold.
IStreamingEventDetectionSession<T> StartStreamingSession() Performs real-time event detection on a streaming session.
IStreamingEventDetectionSession<T> StartStreamingSession(int sampleRate, T threshold) Performs real-time event detection on a streaming session with custom settings.
Task<AudioEventResult<T>> DetectAsync(Tensor<T> audio, CancellationToken cancellationToken) Detects audio events asynchronously.
Tensor<T> GetEventProbabilities(Tensor<T> audio) Gets frame-level event probabilities.