FDYSED<T>

Models & Types · AiDotNet.Audio.Classification

FDY-SED (Frequency Dynamic Sound Event Detection) model for DCASE-winning SED.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → AudioClassifierBase<T> → FDYSED<T>

Implements: IAudioEventDetector<T>

For Beginners

FDY-SED is like having a team of audio specialists:

  • A bass expert identifies low rumbles (washing machines, traffic, thunder)
  • A midrange expert identifies speech, barks, and music
  • A treble expert identifies high-pitched sounds (birds, alarms, glass breaking)

Each expert has their own set of filters optimized for their frequency range, then they combine their findings to produce the final detection.

Usage:

How It Works

FDY-SED (Nam et al., ICASSP 2022) introduces frequency-dynamic convolutions for sound event detection, where convolution kernels are dynamically generated based on the frequency band being processed. Combined with mean-teacher semi-supervised training, it achieves DCASE challenge-winning results on the DESED (Domestic Environment Sound Event Detection) dataset.

Architecture:

  • Frequency-dynamic CNN: 5 CNN blocks with frequency-dynamic convolutions. Each block

generates different kernels for different frequency groups, allowing specialized processing for bass, mid, and treble ranges.

  • Bidirectional GRU: 2 layers of bidirectional GRU for temporal modeling across frames.
  • Frame-level classifier: Sigmoid-activated linear layer for multi-label frame-level detection.

Constructors

Constructor Summary
FDYSED(NeuralNetworkArchitecture<T> architecture, FDYSEDOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a FDYSED for native training mode with the default objective.
FDYSED(NeuralNetworkArchitecture<T> architecture, FDYSEDOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates an FDY-SED model for native training mode.
FDYSED(NeuralNetworkArchitecture<T> architecture, string modelPath, FDYSEDOptions options) Creates a FDYSED for native training mode with the default objective.

Properties

Property Summary
IReadOnlyList<string> EventLabels { get; }
IReadOnlyList<string> SupportedEvents { get; }
double TimeResolution { get; }

Methods

Method Summary
AudioEventResult<T> Detect(Tensor<T> audio)
AudioEventResult<T> Detect(Tensor<T> audio, T threshold)
AudioEventResult<T> DetectSpecific(Tensor<T> audio, IReadOnlyList<string> eventTypes)
AudioEventResult<T> DetectSpecific(Tensor<T> audio, IReadOnlyList<string> eventTypes, T threshold)
IStreamingEventDetectionSession<T> StartStreamingSession()
IStreamingEventDetectionSession<T> StartStreamingSession(int sampleRate, T threshold)
ModelMetadata<T> GetModelMetadata()
Task<AudioEventResult<T>> DetectAsync(Tensor<T> audio, CancellationToken cancellationToken)
Tensor<T> GetEventProbabilities(Tensor<T> audio)

Fields

Field Summary
string[] AudioSetLabels AudioSet-527 standard event labels.