AST<T>

Models & Types · AiDotNet.Audio.Classification

AST (Audio Spectrogram Transformer) model for audio event detection and classification.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → AudioClassifierBase<T> → AST<T>

Implements: IAudioEventDetector<T>

For Beginners

AST is one of the simplest yet effective audio classification models. It works by treating sound spectrograms exactly like images and using a powerful image model (Vision Transformer) to classify them.

Here is how AST processes audio, step by step:

Step 1 - Sound to picture: Audio is converted to a spectrogram (a 2D image showing frequency vs time). This is the same mel spectrogram used by BEATs and other audio models.

Step 2 - Cut into overlapping tiles: The spectrogram is cut into small 16x16 patches with overlap (stride 10), which means each patch shares some pixels with its neighbors. This overlap improves accuracy at the cost of more patches to process.

Step 3 - Add summary tokens: Two special tokens (DeiT’s class and distillation tokens) are added. They collect information from all patches and serve as the overall summary.

Step 4 - Understand context: A 12-layer Transformer encoder lets every patch attend to every other patch. The two summary tokens gather global information.

Step 5 - Classify: The average of the two tokens goes through a linear layer to produce probabilities for each sound class. Sigmoid activation enables multi-label detection.

**Usage with a pre-trained ONNX model (recommended):**Usage with native training:

How It Works

AST (Gong et al., Interspeech 2021) is the first purely attention-based model for audio classification. It directly applies a Vision Transformer (ViT) architecture to audio spectrograms, achieving strong results on multiple benchmarks:

  • AudioSet: 45.9% mAP - competitive with CNN-based models like PANNs
  • ESC-50: 95.6% accuracy - environmental sound classification
  • Speech Commands V2: 98.1% accuracy - keyword spotting

Architecture: AST treats the audio spectrogram as an image and processes it with a standard Vision Transformer:

  • Audio preprocessing: Raw waveform is converted to a 128-bin Kaldi log-mel filterbank (25 ms

frames every 10 ms), padded or cut to 1024 frames and normalized to mean 0 and standard deviation 0.5

  • Patch embedding: The spectrogram is split into overlapping 16x16 patches (stride 10),

each linearly projected to a 768-dimensional embedding

  • Tokens: DeiT’s class and distillation tokens are prepended to the patch sequence
  • Positional encoding: Learnable positional embeddings encode spatial position
  • Transformer encoder: 12 pre-norm blocks with 12-head self-attention
  • Classification: The average of the two tokens’ outputs goes through LayerNorm and a linear head,

and a sigmoid scores each label

Key Innovation: AST demonstrates that ImageNet-pretrained ViT weights transfer effectively to audio spectrograms. By initializing from DeiT (Data-efficient Image Transformer) and fine-tuning on AudioSet, AST achieves competitive results without any audio-specific architectural modifications.

References:

  • Paper: “AST: Audio Spectrogram Transformer” (Gong et al., Interspeech 2021)
  • Repository: https://github.com/YuanGongND/ast

Constructors

Constructor Summary
AST(NeuralNetworkArchitecture<T> architecture, ASTOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates an AST model for native training and inference: the paper’s network, initialized as the official code initializes it when training from scratch.
AST(NeuralNetworkArchitecture<T> architecture, string modelPath, ASTOptions options) Creates an AST model for native training and inference: the paper’s network, initialized as the official code initializes it when training from scratch.

Properties

Property Summary
IReadOnlyList<string> EventLabels { get; }
IReadOnlyList<string> SupportedEvents { get; }
double TimeResolution { get; }

Methods

Method Summary
AudioEventResult<T> Detect(Tensor<T> audio)
AudioEventResult<T> Detect(Tensor<T> audio, T threshold)
AudioEventResult<T> DetectSpecific(Tensor<T> audio, IReadOnlyList<string> eventTypes)
AudioEventResult<T> DetectSpecific(Tensor<T> audio, IReadOnlyList<string> eventTypes, T threshold)
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
IStreamingEventDetectionSession<T> StartStreamingSession()
IStreamingEventDetectionSession<T> StartStreamingSession(int sampleRate, T threshold)
ModelMetadata<T> GetModelMetadata()
Task<AudioEventResult<T>> DetectAsync(Tensor<T> audio, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input)
Tensor<T> GetEventProbabilities(Tensor<T> audio)

Fields

Field Summary
string[] AudioSetLabels AudioSet-527 standard event labels used by AST pre-trained models.