HuBERTSER<T>

Models & Types · AiDotNet.Audio.Emotion

HuBERT-SER (HuBERT for Speech Emotion Recognition) model.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → AudioClassifierBase<T> → HuBERTSER<T>

Implements: IEmotionRecognizer<T>

For Beginners

HuBERT-SER uses a model that first learned to understand speech patterns from millions of hours of audio (HuBERT), then was specialized to detect emotions in voice. It combines deep speech understanding with emotion classification.

Usage:

How It Works

HuBERT-SER fine-tunes the HuBERT (Hsu et al., 2021) self-supervised speech model for emotion recognition. HuBERT learns speech representations through masked prediction of discrete speech units, achieving 69.7% weighted accuracy on IEMOCAP when fine-tuned.

Constructors

Constructor Summary
HuBERTSER(NeuralNetworkArchitecture<T> architecture, HuBERTSEROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a HuBERT-SER model in native training mode.
HuBERTSER(NeuralNetworkArchitecture<T> architecture, string modelPath, HuBERTSEROptions options) Creates a HuBERT-SER model in native training mode.

Properties

Property Summary
IReadOnlyList<string> SupportedEmotions { get; }

Methods

Method Summary
EmotionResult<T> RecognizeEmotion(Tensor<T> audio)
IReadOnlyDictionary<string, T> GetEmotionProbabilities(Tensor<T> audio)
IReadOnlyList<TimedEmotionResult<T>> RecognizeEmotionTimeSeries(Tensor<T> audio, int windowSizeMs, int hopSizeMs)
ModelMetadata<T> GetModelMetadata()
T GetArousal(Tensor<T> audio)
T GetValence(Tensor<T> audio)
Tensor<T> ForwardForTraining(Tensor<T> input) Raw logit forward (no softmax) — used for tape training so the logit loss is well-posed.
Vector<T> ExtractEmotionFeatures(Tensor<T> audio)