HuBERTASR<T>

Models & Types · AiDotNet.SpeechRecognition.Foundation

HuBERT ASR: Hidden-Unit BERT for self-supervised speech representation

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → HuBERTASR<T>

Implements: ISpeechRecognizer<T>

For Beginners

HuBERT uses an offline clustering step to provide pseudo-labels for masked prediction pre-training. Unlike wav2vec 2.0’s contrastive loss, HuBERT predicts discrete cluster assignments of masked speech segments. The model iteratively refines its cl…

How It Works

References:

  • Paper: “HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units” (Hsu et al., Meta, 2021)

HuBERT uses an offline clustering step to provide pseudo-labels for masked prediction pre-training. Unlike wav2vec 2.0’s contrastive loss, HuBERT predicts discrete cluster assignments of masked speech segments. The model iteratively refines its clusters using learned representations from previous iterations. This approach produces rich speech representations that transfer well to ASR, speaker verification, and speech emotion recognition tasks.

Example

// Create a HuBERT ASR model for self-supervised speech representation learning
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.SpeechRecognition,
    inputHeight: 16000, inputWidth: 1, inputDepth: 1, outputSize: 5000);
var model = new HuBERTASR<double>(architecture);
            
// Or load a pre-trained ONNX model for masked prediction ASR
var onnxModel = new HuBERTASR<double>(architecture, "hubertasr.onnx");

Constructors

Constructor Summary
HuBERTASR(NeuralNetworkArchitecture<T> architecture, HuBERTASROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
HuBERTASR(NeuralNetworkArchitecture<T> architecture, string modelPath, HuBERTASROptions options)

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes audio using HuBERT’s masked prediction encoder with CTC decoding.
string DetectLanguage(Tensor<T> audio)
void LoadTorchWeights(Func<string, int[], double[]> read) Loads a transformers *ForCTC state dict straight from source.