EAT<T>
Models & Types · AiDotNet.Audio.Classification
EAT (Efficient Audio Transformer) model for efficient audio event detection and classification.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → AudioClassifierBase<T> → EAT<T>
Implements: IAudioEventDetector<T>
For Beginners
EAT is like having a wise teacher and a student. The teacher sees the full spectrogram and the student only sees 25% of it. The student must predict what the teacher sees. As the student improves, the teacher slowly updates to match, creating a virtuous learning cycle. This is more efficient than BEATs because the student processes fewer patches.
Usage:
How It Works
EAT (Chen et al., 2024) achieves competitive performance with significantly less compute than BEATs through an efficient self-supervised pre-training approach using teacher-student distillation. It reaches 49.7% mAP on AudioSet-2M using only 10% of BEATs’ compute.
Architecture: EAT uses a standard ViT-Base encoder with an EMA teacher:
- Student encoder: 12-layer Transformer that processes visible (unmasked) patches
- Teacher encoder: EMA copy of student that processes all patches, providing targets
- Masked prediction: Student predicts teacher’s representations for masked patches
References:
- Paper: “EAT: Self-Supervised Pre-Training with Efficient Audio Transformer” (Chen et al., 2024)
Constructors
| Constructor | Summary |
|---|---|
EAT(NeuralNetworkArchitecture<T> architecture, EATOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates an EAT model for native training mode. |
EAT(NeuralNetworkArchitecture<T> architecture, string modelPath, EATOptions options) |
Creates an EAT model for native training mode. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> EventLabels { get; } |
|
IReadOnlyList<string> SupportedEvents { get; } |
|
double TimeResolution { get; } |
Methods
| Method | Summary |
|---|---|
AudioEventResult<T> Detect(Tensor<T> audio) |
|
AudioEventResult<T> Detect(Tensor<T> audio, T threshold) |
|
AudioEventResult<T> DetectSpecific(Tensor<T> audio, IReadOnlyList<string> eventTypes) |
|
AudioEventResult<T> DetectSpecific(Tensor<T> audio, IReadOnlyList<string> eventTypes, T threshold) |
|
IStreamingEventDetectionSession<T> StartStreamingSession() |
|
IStreamingEventDetectionSession<T> StartStreamingSession(int sampleRate, T threshold) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<AudioEventResult<T>> DetectAsync(Tensor<T> audio, CancellationToken cancellationToken) |
|
Tensor<T> GetEventProbabilities(Tensor<T> audio) |
Fields
| Field | Summary |
|---|---|
string[] AudioSetLabels |