EAT<T>

Models & Types · AiDotNet.Audio.Classification

EAT (Efficient Audio Transformer) model for efficient audio event detection and classification.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → AudioClassifierBase<T> → EAT<T>

Implements: IAudioEventDetector<T>

For Beginners

EAT is like having a wise teacher and a student. The teacher sees the full spectrogram and the student only sees 25% of it. The student must predict what the teacher sees. As the student improves, the teacher slowly updates to match, creating a virtuous learning cycle. This is more efficient than BEATs because the student processes fewer patches.

Usage:

How It Works

EAT (Chen et al., 2024) achieves competitive performance with significantly less compute than BEATs through an efficient self-supervised pre-training approach using teacher-student distillation. It reaches 49.7% mAP on AudioSet-2M using only 10% of BEATs’ compute.

Architecture: EAT uses a standard ViT-Base encoder with an EMA teacher:

  • Student encoder: 12-layer Transformer that processes visible (unmasked) patches
  • Teacher encoder: EMA copy of student that processes all patches, providing targets
  • Masked prediction: Student predicts teacher’s representations for masked patches

References:

  • Paper: “EAT: Self-Supervised Pre-Training with Efficient Audio Transformer” (Chen et al., 2024)

Constructors

Constructor Summary
EAT(NeuralNetworkArchitecture<T> architecture, EATOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates an EAT model for native training mode.
EAT(NeuralNetworkArchitecture<T> architecture, string modelPath, EATOptions options) Creates an EAT model for native training mode.

Properties

Property Summary
IReadOnlyList<string> EventLabels { get; }
IReadOnlyList<string> SupportedEvents { get; }
double TimeResolution { get; }

Methods

Method Summary
AudioEventResult<T> Detect(Tensor<T> audio)
AudioEventResult<T> Detect(Tensor<T> audio, T threshold)
AudioEventResult<T> DetectSpecific(Tensor<T> audio, IReadOnlyList<string> eventTypes)
AudioEventResult<T> DetectSpecific(Tensor<T> audio, IReadOnlyList<string> eventTypes, T threshold)
IStreamingEventDetectionSession<T> StartStreamingSession()
IStreamingEventDetectionSession<T> StartStreamingSession(int sampleRate, T threshold)
ModelMetadata<T> GetModelMetadata()
Task<AudioEventResult<T>> DetectAsync(Tensor<T> audio, CancellationToken cancellationToken)
Tensor<T> GetEventProbabilities(Tensor<T> audio)

Fields

Field Summary
string[] AudioSetLabels