FDYSED<T>
Models & Types · AiDotNet.Audio.Classification
FDY-SED (Frequency Dynamic Sound Event Detection) model for DCASE-winning SED.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → AudioClassifierBase<T> → FDYSED<T>
Implements: IAudioEventDetector<T>
For Beginners
FDY-SED is like having a team of audio specialists:
- A bass expert identifies low rumbles (washing machines, traffic, thunder)
- A midrange expert identifies speech, barks, and music
- A treble expert identifies high-pitched sounds (birds, alarms, glass breaking)
Each expert has their own set of filters optimized for their frequency range, then they combine their findings to produce the final detection.
Usage:
How It Works
FDY-SED (Nam et al., ICASSP 2022) introduces frequency-dynamic convolutions for sound event detection, where convolution kernels are dynamically generated based on the frequency band being processed. Combined with mean-teacher semi-supervised training, it achieves DCASE challenge-winning results on the DESED (Domestic Environment Sound Event Detection) dataset.
Architecture:
- Frequency-dynamic CNN: 5 CNN blocks with frequency-dynamic convolutions. Each block
generates different kernels for different frequency groups, allowing specialized processing for bass, mid, and treble ranges.
- Bidirectional GRU: 2 layers of bidirectional GRU for temporal modeling across frames.
- Frame-level classifier: Sigmoid-activated linear layer for multi-label frame-level detection.
Constructors
| Constructor | Summary |
|---|---|
FDYSED(NeuralNetworkArchitecture<T> architecture, FDYSEDOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a FDYSED for native training mode with the default objective. |
FDYSED(NeuralNetworkArchitecture<T> architecture, FDYSEDOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
Creates an FDY-SED model for native training mode. |
FDYSED(NeuralNetworkArchitecture<T> architecture, string modelPath, FDYSEDOptions options) |
Creates a FDYSED for native training mode with the default objective. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> EventLabels { get; } |
|
IReadOnlyList<string> SupportedEvents { get; } |
|
double TimeResolution { get; } |
Methods
| Method | Summary |
|---|---|
AudioEventResult<T> Detect(Tensor<T> audio) |
|
AudioEventResult<T> Detect(Tensor<T> audio, T threshold) |
|
AudioEventResult<T> DetectSpecific(Tensor<T> audio, IReadOnlyList<string> eventTypes) |
|
AudioEventResult<T> DetectSpecific(Tensor<T> audio, IReadOnlyList<string> eventTypes, T threshold) |
|
IStreamingEventDetectionSession<T> StartStreamingSession() |
|
IStreamingEventDetectionSession<T> StartStreamingSession(int sampleRate, T threshold) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<AudioEventResult<T>> DetectAsync(Tensor<T> audio, CancellationToken cancellationToken) |
|
Tensor<T> GetEventProbabilities(Tensor<T> audio) |
Fields
| Field | Summary |
|---|---|
string[] AudioSetLabels |
AudioSet-527 standard event labels. |