HuBERT<T>
Models & Types · AiDotNet.Audio.Foundations
HuBERT (Hidden-Unit BERT) self-supervised speech representation model.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → HuBERT<T>
Implements: IAudioFoundationModel<T>, IPaperOptimizerVariant
For Beginners
HuBERT is a foundation model for audio - it understands speech at a deep level without needing any labels during training. It can be used as a feature extractor: feed it audio and get rich embeddings that capture everything about the speech (what’s said, who’s speaking, their emotion, etc.).
Usage:
How It Works
HuBERT (Hsu et al., 2021, Meta) learns speech representations by predicting masked discrete speech units. It uses an offline clustering step to create pseudo-labels from audio, then trains a BERT-like masked prediction task. HuBERT-Base (12 layers) and HuBERT-Large (24 layers) achieve state-of-the-art on multiple speech processing tasks.
Constructors
| Constructor | Summary |
|---|---|
HuBERT(NeuralNetworkArchitecture<T> architecture, HuBERTOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a HuBERT model in native training mode. |
HuBERT(NeuralNetworkArchitecture<T> architecture, string modelPath, HuBERTOptions options) |
Creates a HuBERT model in native training mode. |
Properties
| Property | Summary |
|---|---|
int EmbeddingDimension { get; } |
|
int NumLayers { get; } |
|
string PaperOptimizerVariant { get; } |
The variant this instance was built at, keying the paper recipe for it. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Task<Tensor<T>> ExtractEmbeddingsAsync(Tensor<T> audio, CancellationToken cancellationToken) |
|
Tensor<T> ExtractEmbeddings(Tensor<T> audio) |
|
Tensor<T> ExtractLayerFeatures(Tensor<T> audio, int layerIndex) |
|
Tensor<T> ExtractWeightedFeatures(Tensor<T> audio, T[] layerWeights) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
void LoadTorchWeights(Func<string, int[], double[]> read) |
Loads a transformers HubertModel state dict straight from source. |