ASTOptions

Options & Configuration · AiDotNet.Audio.Classification

Configuration options for the AST (Audio Spectrogram Transformer) models, AST and ASTModel, which run one network.

Inheritance: ModelOptions → ASTOptions

For Beginners

AST treats an audio spectrogram as an image. It cuts the spectrogram into overlapping patches, then uses a Transformer to relate them, and scores every sound label independently.

Example usage:

How It Works

AST (Gong, Chung and Glass, Interspeech 2021) applies a Vision Transformer to audio. The defaults are the paper’s AudioSet configuration: 128-bin Kaldi filterbanks of 10-second clips (1024 frames), 16×16 patches with an overlap of 6, the 768-wide 12-layer DeiT body with the class and distillation tokens averaged, inputs normalized to mean 0 and standard deviation 0.5, and training with mixup and SpecAugment.

References:

  • Paper: “AST: Audio Spectrogram Transformer” (Gong et al., Interspeech 2021)
  • Repository: https://github.com/YuanGongND/ast

Constructors

Constructor Summary
ASTOptions() Creates an instance with the default settings.
ASTOptions(ASTOptions other) Copies the settings from another options instance.

Properties

Property Summary
OnnxModelOptions OnnxOptions { get; set; } Gets or sets ONNX runtime options.
WindowType Window { get; set; } Gets or sets the filterbank’s analysis window.
double DatasetMean { get; set; } Gets or sets the dataset filterbank mean the input is normalized by (AudioSet’s, from the official recipe).
double DatasetStd { get; set; } Gets or sets the dataset filterbank standard deviation (AudioSet’s).
double MaxGradNorm { get; set; } Gets or sets the training gradient norm limit.
double MixupRate { get; set; } Gets or sets the probability that a training clip is mixed with another (0.5, §3.1.1).
double Threshold { get; set; } Gets or sets the confidence threshold for event detection.
double WindowOverlap { get; set; } Gets or sets the window overlap ratio (0-1) for event detection.
double WindowSize { get; set; } Gets or sets the window size in seconds for event detection.
int EmbeddingDim { get; set; } Gets or sets the Transformer width (768).
int FeedForwardDim { get; set; } Gets or sets the MLP width of each block (3072, four times the width).
int FrequencyMaskLength { get; set; } Gets or sets the maximum frequency-mask length in mel bins (48, §3.1.1).
int FrequencyStride { get; set; } Gets or sets the patch stride along frequency (10: an overlap of 6, §2.1).
int NumAttentionHeads { get; set; } Gets or sets the number of attention heads (12).
int NumEncoderLayers { get; set; } Gets or sets the number of Transformer blocks (12).
int NumMels { get; set; } Gets or sets the number of mel bins of the Kaldi filterbank (128, §2.1).
int PatchSize { get; set; } Gets or sets the square patch size (16).
int SampleRate { get; set; } Gets or sets the audio sample rate in Hz (16 kHz in the paper).
int TargetLength { get; set; } Gets or sets the number of filterbank frames the network takes (1024: 10 s at the 10 ms frame shift).
int TimeMaskLength { get; set; } Gets or sets the maximum time-mask length in frames (192, §3.1.1).
int TimeStride { get; set; } Gets or sets the patch stride along time (10).
string ModelPath { get; set; } Gets or sets the path to a pre-trained ONNX model file.
string[] CustomLabels { get; set; } Gets or sets custom event labels.