AudioMAE<T>

Models & Types · AiDotNet.Audio.Classification

Audio-MAE (Masked Autoencoders for Audio) model for audio classification.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → AudioClassifierBase<T> → AudioMAE<T>

Implements: IAudioEventDetector<T>

For Beginners

Audio-MAE learns by hiding 80% of a spectrogram and trying to reconstruct it. This “fill in the blanks” approach forces the model to deeply understand audio patterns. After pre-training, only the encoder is kept for classification.

Usage:

How It Works

Audio-MAE (Xu et al., NeurIPS 2022) applies Masked Autoencoders to audio spectrograms. By masking 80% of spectrogram patches and reconstructing them, Audio-MAE learns rich audio representations achieving 47.3% mAP on AudioSet and 97.0% on ESC-50.

References:

  • Paper: “Masked Autoencoders that Listen” (Xu et al., NeurIPS 2022)
  • Repository: https://github.com/facebookresearch/AudioMAE

Constructors

Constructor Summary
AudioMAE(NeuralNetworkArchitecture<T> architecture, AudioMAEOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates an Audio-MAE model for native training mode.
AudioMAE(NeuralNetworkArchitecture<T> architecture, string modelPath, AudioMAEOptions options) Creates an Audio-MAE model for native training mode.

Properties

Property Summary
IReadOnlyList<string> EventLabels { get; }
IReadOnlyList<string> SupportedEvents { get; }
double TimeResolution { get; }

Methods

Method Summary
AudioEventResult<T> Detect(Tensor<T> audio)
AudioEventResult<T> Detect(Tensor<T> audio, T threshold)
AudioEventResult<T> DetectSpecific(Tensor<T> audio, IReadOnlyList<string> eventTypes)
AudioEventResult<T> DetectSpecific(Tensor<T> audio, IReadOnlyList<string> eventTypes, T threshold)
IStreamingEventDetectionSession<T> StartStreamingSession()
IStreamingEventDetectionSession<T> StartStreamingSession(int sampleRate, T threshold)
ModelMetadata<T> GetModelMetadata()
Task<AudioEventResult<T>> DetectAsync(Tensor<T> audio, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input) Training forward — the same encoder → average-pool → linear path as inference.
Tensor<T> GetEventProbabilities(Tensor<T> audio)

Fields

Field Summary
string[] AudioSetLabels