AudioMAE<T>
Models & Types · AiDotNet.Audio.Classification
Audio-MAE (Masked Autoencoders for Audio) model for audio classification.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → AudioClassifierBase<T> → AudioMAE<T>
Implements: IAudioEventDetector<T>
For Beginners
Audio-MAE learns by hiding 80% of a spectrogram and trying to reconstruct it. This “fill in the blanks” approach forces the model to deeply understand audio patterns. After pre-training, only the encoder is kept for classification.
Usage:
How It Works
Audio-MAE (Xu et al., NeurIPS 2022) applies Masked Autoencoders to audio spectrograms. By masking 80% of spectrogram patches and reconstructing them, Audio-MAE learns rich audio representations achieving 47.3% mAP on AudioSet and 97.0% on ESC-50.
References:
- Paper: “Masked Autoencoders that Listen” (Xu et al., NeurIPS 2022)
- Repository:
https://github.com/facebookresearch/AudioMAE
Constructors
| Constructor | Summary |
|---|---|
AudioMAE(NeuralNetworkArchitecture<T> architecture, AudioMAEOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates an Audio-MAE model for native training mode. |
AudioMAE(NeuralNetworkArchitecture<T> architecture, string modelPath, AudioMAEOptions options) |
Creates an Audio-MAE model for native training mode. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> EventLabels { get; } |
|
IReadOnlyList<string> SupportedEvents { get; } |
|
double TimeResolution { get; } |
Methods
| Method | Summary |
|---|---|
AudioEventResult<T> Detect(Tensor<T> audio) |
|
AudioEventResult<T> Detect(Tensor<T> audio, T threshold) |
|
AudioEventResult<T> DetectSpecific(Tensor<T> audio, IReadOnlyList<string> eventTypes) |
|
AudioEventResult<T> DetectSpecific(Tensor<T> audio, IReadOnlyList<string> eventTypes, T threshold) |
|
IStreamingEventDetectionSession<T> StartStreamingSession() |
|
IStreamingEventDetectionSession<T> StartStreamingSession(int sampleRate, T threshold) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<AudioEventResult<T>> DetectAsync(Tensor<T> audio, CancellationToken cancellationToken) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
Training forward — the same encoder → average-pool → linear path as inference. |
Tensor<T> GetEventProbabilities(Tensor<T> audio) |
Fields
| Field | Summary |
|---|---|
string[] AudioSetLabels |