HuBERTASR<T>
Models & Types · AiDotNet.SpeechRecognition.Foundation
HuBERT ASR: Hidden-Unit BERT for self-supervised speech representation
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → HuBERTASR<T>
Implements: ISpeechRecognizer<T>
For Beginners
HuBERT uses an offline clustering step to provide pseudo-labels for masked prediction pre-training. Unlike wav2vec 2.0’s contrastive loss, HuBERT predicts discrete cluster assignments of masked speech segments. The model iteratively refines its cl…
How It Works
References:
- Paper: “HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units” (Hsu et al., Meta, 2021)
HuBERT uses an offline clustering step to provide pseudo-labels for masked prediction pre-training. Unlike wav2vec 2.0’s contrastive loss, HuBERT predicts discrete cluster assignments of masked speech segments. The model iteratively refines its clusters using learned representations from previous iterations. This approach produces rich speech representations that transfer well to ASR, speaker verification, and speech emotion recognition tasks.
Example
// Create a HuBERT ASR model for self-supervised speech representation learning
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.SpeechRecognition,
inputHeight: 16000, inputWidth: 1, inputDepth: 1, outputSize: 5000);
var model = new HuBERTASR<double>(architecture);
// Or load a pre-trained ONNX model for masked prediction ASR
var onnxModel = new HuBERTASR<double>(architecture, "hubertasr.onnx");
Constructors
| Constructor | Summary |
|---|---|
HuBERTASR(NeuralNetworkArchitecture<T> architecture, HuBERTASROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
HuBERTASR(NeuralNetworkArchitecture<T> architecture, string modelPath, HuBERTASROptions options) |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
|
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
Transcribes audio using HuBERT’s masked prediction encoder with CTC decoding. |
string DetectLanguage(Tensor<T> audio) |
|
void LoadTorchWeights(Func<string, int[], double[]> read) |
Loads a transformers *ForCTC state dict straight from source. |