HuBERT<T>

Models & Types · AiDotNet.Audio.Foundations

HuBERT (Hidden-Unit BERT) self-supervised speech representation model.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → HuBERT<T>

Implements: IAudioFoundationModel<T>, IPaperOptimizerVariant

For Beginners

HuBERT is a foundation model for audio - it understands speech at a deep level without needing any labels during training. It can be used as a feature extractor: feed it audio and get rich embeddings that capture everything about the speech (what’s said, who’s speaking, their emotion, etc.).

Usage:

How It Works

HuBERT (Hsu et al., 2021, Meta) learns speech representations by predicting masked discrete speech units. It uses an offline clustering step to create pseudo-labels from audio, then trains a BERT-like masked prediction task. HuBERT-Base (12 layers) and HuBERT-Large (24 layers) achieve state-of-the-art on multiple speech processing tasks.

Constructors

Constructor Summary
HuBERT(NeuralNetworkArchitecture<T> architecture, HuBERTOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a HuBERT model in native training mode.
HuBERT(NeuralNetworkArchitecture<T> architecture, string modelPath, HuBERTOptions options) Creates a HuBERT model in native training mode.

Properties

Property Summary
int EmbeddingDimension { get; }
int NumLayers { get; }
string PaperOptimizerVariant { get; } The variant this instance was built at, keying the paper recipe for it.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Task<Tensor<T>> ExtractEmbeddingsAsync(Tensor<T> audio, CancellationToken cancellationToken)
Tensor<T> ExtractEmbeddings(Tensor<T> audio)
Tensor<T> ExtractLayerFeatures(Tensor<T> audio, int layerIndex)
Tensor<T> ExtractWeightedFeatures(Tensor<T> audio, T[] layerWeights)
Tensor<T> ForwardForTraining(Tensor<T> input)
void LoadTorchWeights(Func<string, int[], double[]> read) Loads a transformers HubertModel state dict straight from source.