HTSAT<T>
Models & Types · AiDotNet.Audio.Classification
HTS-AT (Hierarchical Token-Semantic Audio Transformer) model for efficient audio classification.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → AudioClassifierBase<T> → HTSAT<T>
Implements: IAudioEventDetector<T>
For Beginners
HTS-AT is like reading a book at different zoom levels. First it reads individual words (small patches), then sentences (merged patches), then paragraphs (further merged), and finally understands the whole story. At each level, it uses “window attention” - looking at nearby patches first, which is much faster than looking at everything.
Usage with ONNX (recommended):
How It Works
HTS-AT (Chen et al., ICASSP 2022) is a hierarchical Transformer architecture that uses Swin Transformer blocks with a token-semantic module for efficient audio classification. It achieves 47.1% mAP on AudioSet-2M with only 30M parameters, making it more parameter-efficient than models like AST (87M parameters) while achieving higher accuracy.
Architecture: HTS-AT processes audio spectrograms hierarchically through four stages:
- Patch embedding: 4x4 patch embedding of the mel spectrogram
- Stage 1: 2 Swin blocks at 96-dim with local window attention
- Stage 2: 2 Swin blocks at 192-dim (patch merging doubles channels, halves resolution)
- Stage 3: 6 Swin blocks at 384-dim (the deepest processing stage)
- Stage 4: 2 Swin blocks at 768-dim
- Token-semantic module: Groups tokens by semantic meaning for global context
- Classification head: Projects to class logits with sigmoid for multi-label
References:
- Paper: “HTS-AT: A Hierarchical Token-Semantic Audio Transformer” (Chen et al., ICASSP 2022)
- Repository:
https://github.com/RetroCirce/HTS-Audio-Transformer
Constructors
| Constructor | Summary |
|---|---|
HTSAT(NeuralNetworkArchitecture<T> architecture, HTSATOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates an HTS-AT model for native training mode. |
HTSAT(NeuralNetworkArchitecture<T> architecture, string modelPath, HTSATOptions options) |
Creates an HTS-AT model for native training mode. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> EventLabels { get; } |
|
IReadOnlyList<string> SupportedEvents { get; } |
|
double TimeResolution { get; } |
Methods
| Method | Summary |
|---|---|
AudioEventResult<T> Detect(Tensor<T> audio) |
|
AudioEventResult<T> Detect(Tensor<T> audio, T threshold) |
|
AudioEventResult<T> DetectSpecific(Tensor<T> audio, IReadOnlyList<string> eventTypes) |
|
AudioEventResult<T> DetectSpecific(Tensor<T> audio, IReadOnlyList<string> eventTypes, T threshold) |
|
IStreamingEventDetectionSession<T> StartStreamingSession() |
|
IStreamingEventDetectionSession<T> StartStreamingSession(int sampleRate, T threshold) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<AudioEventResult<T>> DetectAsync(Tensor<T> audio, CancellationToken cancellationToken) |
|
Tensor<T> GetEventProbabilities(Tensor<T> audio) |
Fields
| Field | Summary |
|---|---|
string[] AudioSetLabels |
AudioSet-527 standard event labels. |