ASTModel<T>

Models & Types · AiDotNet.Audio.Fingerprinting

AST (Audio Spectrogram Transformer, Gong et al. 2021) as an audio fingerprinter: the clip’s AST representation is its fingerprint, compared by cosine similarity.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → ASTModel<T>

Implements: IAudioFingerprinter<T>

How It Works

This runs the same network as AST (AstNetwork, the official code’s model): a 128-bin Kaldi filterbank of the waveform, 16×16 patches with an overlap of 6, DeiT’s class and distillation tokens, 12 pre-norm Transformer blocks, and the average of the two tokens as the clip representation. The fingerprint is that representation; Classify scores the labels with the paper’s sigmoid head, and training is the paper’s AudioSet recipe.

Reference: Gong, Y., Chung, Y.-A. & Glass, J. (2021), “AST: Audio Spectrogram Transformer”, Interspeech.

Constructors

Constructor Summary
ASTModel(NeuralNetworkArchitecture<T> architecture) Calls the fully specified overload with the omitted parameters’ defaults.
ASTModel(NeuralNetworkArchitecture<T> architecture, ASTOptions options) Calls the fully specified overload with the omitted parameters’ defaults.
ASTModel(NeuralNetworkArchitecture<T> architecture, ASTOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Initializes AST in native training / inference mode.
ASTModel(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Calls the fully specified overload with the omitted parameters’ defaults.
ASTModel(NeuralNetworkArchitecture<T> architecture, string modelPath) Calls the fully specified overload with the omitted parameters’ defaults.
ASTModel(NeuralNetworkArchitecture<T> architecture, string modelPath, ASTOptions options) Initializes AST in native training / inference mode.

Properties

Property Summary
IReadOnlyList<string> Labels { get; } The labels Classify names (CustomLabels, else AudioSet’s 527).
int FingerprintLength { get; }
string Name { get; }

Methods

Method Summary
AudioFingerprint<T> Fingerprint(Tensor<T> audio)
AudioFingerprint<T> Fingerprint(Vector<T> audio)
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
IReadOnlyList<FingerprintMatch> FindMatches(AudioFingerprint<T> query, AudioFingerprint<T> reference, int minMatchLength)
List<ValueTuple<string, double>> Classify(Tensor<T> audio, int topK) The topK most probable labels of the first clip, by the paper’s independent sigmoid scores (AudioSet is multi-label).
ModelMetadata<T> GetModelMetadata()
Tensor<T> ForwardForTraining(Tensor<T> input)
double ComputeSimilarity(AudioFingerprint<T> fp1, AudioFingerprint<T> fp2)