IAudioFoundationModel<T>

Interfaces · AiDotNet.Interfaces

Defines the contract for self-supervised audio foundation models.

Known implementations: Data2Vec2<T>, HuBERT<T>, MERT<T>, W2vBert<T>, Wav2Vec2<T>, WavLM<T>

For Beginners

A foundation model is like a student who has read millions of books but hasn’t been told what to look for. It develops a deep understanding of language (or in this case, audio). When you need it for a specific task (like recognizing emotions), you just teach it the final step - it already understands the audio.

These models provide embeddings (numerical representations) that capture:

  • Phonetic content (what sounds are being made)
  • Speaker characteristics (who is speaking)
  • Prosody and emotion (how they’re speaking)
  • Acoustic environment (where the recording was made)

Examples: HuBERT, wav2vec 2.0, WavLM, data2vec

How It Works

Audio foundation models learn general-purpose audio representations through self-supervised pre-training on large unlabeled datasets. These representations can be fine-tuned for downstream tasks like speech recognition, speaker verification, and emotion detection.

Properties

Property Summary
int EmbeddingDimension { get; } Gets the embedding dimension of the model.
int NumLayers { get; } Gets the number of transformer layers in the model.
int SampleRate { get; } Gets the sample rate this model operates at.

Methods

Method Summary
Task<Tensor<T>> ExtractEmbeddingsAsync(Tensor<T> audio, CancellationToken cancellationToken) Extracts embeddings asynchronously.
Tensor<T> ExtractEmbeddings(Tensor<T> audio) Extracts the final-layer embeddings from audio.
Tensor<T> ExtractLayerFeatures(Tensor<T> audio, int layerIndex) Extracts features from a specific layer.
Tensor<T> ExtractWeightedFeatures(Tensor<T> audio, T[] layerWeights) Extracts weighted combination of features from all layers.