IAudioFoundationModel<T>
Interfaces · AiDotNet.Interfaces
Defines the contract for self-supervised audio foundation models.
Known implementations: Data2Vec2<T>, HuBERT<T>, MERT<T>, W2vBert<T>, Wav2Vec2<T>, WavLM<T>
For Beginners
A foundation model is like a student who has read millions of books but hasn’t been told what to look for. It develops a deep understanding of language (or in this case, audio). When you need it for a specific task (like recognizing emotions), you just teach it the final step - it already understands the audio.
These models provide embeddings (numerical representations) that capture:
- Phonetic content (what sounds are being made)
- Speaker characteristics (who is speaking)
- Prosody and emotion (how they’re speaking)
- Acoustic environment (where the recording was made)
Examples: HuBERT, wav2vec 2.0, WavLM, data2vec
How It Works
Audio foundation models learn general-purpose audio representations through self-supervised pre-training on large unlabeled datasets. These representations can be fine-tuned for downstream tasks like speech recognition, speaker verification, and emotion detection.
Properties
| Property | Summary |
|---|---|
int EmbeddingDimension { get; } |
Gets the embedding dimension of the model. |
int NumLayers { get; } |
Gets the number of transformer layers in the model. |
int SampleRate { get; } |
Gets the sample rate this model operates at. |
Methods
| Method | Summary |
|---|---|
Task<Tensor<T>> ExtractEmbeddingsAsync(Tensor<T> audio, CancellationToken cancellationToken) |
Extracts embeddings asynchronously. |
Tensor<T> ExtractEmbeddings(Tensor<T> audio) |
Extracts the final-layer embeddings from audio. |
Tensor<T> ExtractLayerFeatures(Tensor<T> audio, int layerIndex) |
Extracts features from a specific layer. |
Tensor<T> ExtractWeightedFeatures(Tensor<T> audio, T[] layerWeights) |
Extracts weighted combination of features from all layers. |