HuBERTSER<T>
Models & Types · AiDotNet.Audio.Emotion
HuBERT-SER (HuBERT for Speech Emotion Recognition) model.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → AudioClassifierBase<T> → HuBERTSER<T>
Implements: IEmotionRecognizer<T>
For Beginners
HuBERT-SER uses a model that first learned to understand speech patterns from millions of hours of audio (HuBERT), then was specialized to detect emotions in voice. It combines deep speech understanding with emotion classification.
Usage:
How It Works
HuBERT-SER fine-tunes the HuBERT (Hsu et al., 2021) self-supervised speech model for emotion recognition. HuBERT learns speech representations through masked prediction of discrete speech units, achieving 69.7% weighted accuracy on IEMOCAP when fine-tuned.
Constructors
| Constructor | Summary |
|---|---|
HuBERTSER(NeuralNetworkArchitecture<T> architecture, HuBERTSEROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a HuBERT-SER model in native training mode. |
HuBERTSER(NeuralNetworkArchitecture<T> architecture, string modelPath, HuBERTSEROptions options) |
Creates a HuBERT-SER model in native training mode. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedEmotions { get; } |
Methods
| Method | Summary |
|---|---|
EmotionResult<T> RecognizeEmotion(Tensor<T> audio) |
|
IReadOnlyDictionary<string, T> GetEmotionProbabilities(Tensor<T> audio) |
|
IReadOnlyList<TimedEmotionResult<T>> RecognizeEmotionTimeSeries(Tensor<T> audio, int windowSizeMs, int hopSizeMs) |
|
ModelMetadata<T> GetModelMetadata() |
|
T GetArousal(Tensor<T> audio) |
|
T GetValence(Tensor<T> audio) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
Raw logit forward (no softmax) — used for tape training so the logit loss is well-posed. |
Vector<T> ExtractEmotionFeatures(Tensor<T> audio) |