IEmotionRecognizer<T>

Interfaces · AiDotNet.Interfaces

Defines the contract for speech emotion recognition models.

Known implementations: Emotion2Vec<T>, HuBERTSER<T>, SpeechEmotionRecognizer<T>, Wav2Small<T>

For Beginners

This is like reading emotions from someone’s voice!

How humans convey emotion in speech:

  • Pitch: Higher when excited/happy, lower when sad
  • Speed: Faster when angry/excited, slower when sad
  • Volume: Louder when angry, softer when sad/fearful
  • Voice quality: Breathy, tense, relaxed

Applications:

  • Call centers: Detect frustrated customers for escalation
  • Mental health: Monitor patient emotional state
  • Voice assistants: Respond appropriately to user mood
  • Gaming: Adapt game difficulty/story based on player emotion
  • Market research: Analyze focus group reactions

Challenges:

  • Cultural differences in emotional expression
  • Speaker variability (age, gender, accent)
  • Context dependency (same words can mean different emotions)
  • Mixed emotions (happy but nervous)

How It Works

Speech Emotion Recognition (SER) identifies emotional states from voice:

  • Basic emotions: Happy, Sad, Angry, Fear, Surprise, Disgust, Neutral
  • Arousal: Low (calm) to High (excited)
  • Valence: Negative to Positive
  • Dominance: Submissive to Dominant

Properties

Property Summary
IReadOnlyList<string> SupportedEmotions { get; } Gets the list of emotions this model can detect.
int SampleRate { get; } Gets the sample rate this recognizer operates at.

Methods

Method Summary
EmotionResult<T> RecognizeEmotion(Tensor<T> audio) Recognizes the primary emotion in speech audio.
IReadOnlyDictionary<string, T> GetEmotionProbabilities(Tensor<T> audio) Gets probabilities for all supported emotions.
IReadOnlyList<TimedEmotionResult<T>> RecognizeEmotionTimeSeries(Tensor<T> audio, int windowSizeMs, int hopSizeMs) Recognizes emotions over time (for longer recordings).
T GetArousal(Tensor<T> audio) Gets arousal (activation) level from speech.
T GetValence(Tensor<T> audio) Gets valence (positivity) level from speech.
Vector<T> ExtractEmotionFeatures(Tensor<T> audio) Extracts emotion-relevant features from audio.