AudioEventDetector<T>
Models & Types · AiDotNet.Audio.Classification
Audio event detection model for identifying sounds in audio (AudioSet-style).
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → AudioClassifierBase<T> → AudioEventDetector<T>
Implements: IAudioEventDetector<T>
For Beginners
Audio event detection answers “What sounds are in this audio?”:
- Human sounds: speech, laughter, coughing, footsteps
- Animal sounds: dog barking, bird singing, cat meowing
- Music: instruments, genres, singing
- Environmental: traffic, rain, wind, construction
Usage with ONNX:
Usage with training:
How It Works
Detects various audio events like speech, music, environmental sounds, and more. Based on AudioSet ontology with 527+ event classes organized hierarchically.
Architecture: Hershey et al. 2017’s best classifier, ResNet-50 adapted to audio (section 3.3.5): the stride of 2 is removed from the first 7x7 convolution and the final average pool covers the 6 x 4 activations of a 96 x 64 log-mel patch (960 ms of 25 ms windows every 10 ms, 64 mel bins, log after a small offset). Batch normalization follows every convolution and one sigmoid unit per label replaces the softmax, trained with cross-entropy and Adam (section 3.1). Audio is classified in non-overlapping 960 ms windows.
Constructors
| Constructor | Summary |
|---|---|
AudioEventDetector(AudioEventDetectorOptions options) |
Creates an AudioEventDetector with legacy options only (native mode). |
AudioEventDetector(NeuralNetworkArchitecture<T> architecture, AudioEventDetectorOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates an AudioEventDetector for native training mode with the default objective. |
AudioEventDetector(NeuralNetworkArchitecture<T> architecture, AudioEventDetectorOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
Creates an AudioEventDetector for native training mode. |
AudioEventDetector(NeuralNetworkArchitecture<T> architecture, string modelPath, AudioEventDetectorOptions options) |
Creates an AudioEventDetector for native training mode with the default objective. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> EventLabels { get; } |
Gets the event labels (alias for SupportedEvents for legacy API compatibility). |
IReadOnlyList<string> SupportedEvents { get; } |
Gets the list of event types this model can detect. |
double TimeResolution { get; } |
Gets the time resolution for event detection in seconds. |
Methods
| Method | Summary |
|---|---|
AudioEventResult<T> Detect(Tensor<T> audio) |
Detects audio events in the audio stream. |
AudioEventResult<T> Detect(Tensor<T> audio, T threshold) |
Detects audio events with custom threshold. |
AudioEventResult<T> DetectSpecific(Tensor<T> audio, IReadOnlyList<string> eventTypes) |
Detects specific events only. |
AudioEventResult<T> DetectSpecific(Tensor<T> audio, IReadOnlyList<string> eventTypes, T threshold) |
Detects specific events only with custom threshold. |
Dictionary<string, double> DetectFrame(Tensor<T> audio) |
Detects a single frame (no windowing) - legacy API. |
IStreamingEventDetectionSession<T> StartStreamingSession() |
Starts a streaming event detection session. |
IStreamingEventDetectionSession<T> StartStreamingSession(int sampleRate, T threshold) |
Starts a streaming event detection session with custom settings. |
List<AudioEvent> DetectLegacy(Tensor<T> audio) |
Detects audio events in the given audio (legacy API). |
List<ValueTuple<string, double>> DetectTopK(Tensor<T> audio, int topK) |
Gets the top K events for a single frame (legacy API). |
ModelMetadata<T> GetModelMetadata() |
Gets model metadata. |
Task<AudioEventResult<T>> DetectAsync(Tensor<T> audio, CancellationToken cancellationToken) |
Detects audio events asynchronously. |
Task<List<AudioEvent>> DetectLegacyAsync(Tensor<T> audio, CancellationToken cancellationToken) |
Detects audio events asynchronously (legacy API). |
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
Tensor<T> GetEventProbabilities(Tensor<T> audio) |
Gets frame-level event probabilities. |
Fields
| Field | Summary |
|---|---|
string[] CommonEventLabels |
Common audio event categories from AudioSet. |