ASTModel<T>
Models & Types · AiDotNet.Audio.Fingerprinting
AST (Audio Spectrogram Transformer, Gong et al. 2021) as an audio fingerprinter: the clip’s AST representation is its fingerprint, compared by cosine similarity.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → ASTModel<T>
Implements: IAudioFingerprinter<T>
How It Works
This runs the same network as AST (AstNetwork, the official code’s model): a
128-bin Kaldi filterbank of the waveform, 16×16 patches with an overlap of 6, DeiT’s class and distillation tokens,
12 pre-norm Transformer blocks, and the average of the two tokens as the clip representation. The fingerprint is
that representation; Classify scores the labels with the paper’s sigmoid head, and training is the
paper’s AudioSet recipe.
Reference: Gong, Y., Chung, Y.-A. & Glass, J. (2021), “AST: Audio Spectrogram Transformer”, Interspeech.
Constructors
| Constructor | Summary |
|---|---|
ASTModel(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
ASTModel(NeuralNetworkArchitecture<T> architecture, ASTOptions options) |
Calls the fully specified overload with the omitted parameters’ defaults. |
ASTModel(NeuralNetworkArchitecture<T> architecture, ASTOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Initializes AST in native training / inference mode. |
ASTModel(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Calls the fully specified overload with the omitted parameters’ defaults. |
ASTModel(NeuralNetworkArchitecture<T> architecture, string modelPath) |
Calls the fully specified overload with the omitted parameters’ defaults. |
ASTModel(NeuralNetworkArchitecture<T> architecture, string modelPath, ASTOptions options) |
Initializes AST in native training / inference mode. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> Labels { get; } |
The labels Classify names (CustomLabels, else AudioSet’s 527). |
int FingerprintLength { get; } |
|
string Name { get; } |
Methods
| Method | Summary |
|---|---|
AudioFingerprint<T> Fingerprint(Tensor<T> audio) |
|
AudioFingerprint<T> Fingerprint(Vector<T> audio) |
|
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) |
|
IReadOnlyList<FingerprintMatch> FindMatches(AudioFingerprint<T> query, AudioFingerprint<T> reference, int minMatchLength) |
|
List<ValueTuple<string, double>> Classify(Tensor<T> audio, int topK) |
The topK most probable labels of the first clip, by the paper’s independent sigmoid scores (AudioSet is multi-label). |
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
double ComputeSimilarity(AudioFingerprint<T> fp1, AudioFingerprint<T> fp2) |