SpeechEmotionRecognizer<T>

Models & Types · AiDotNet.Audio.Emotion

Neural network-based speech emotion recognition model that classifies emotional states from audio.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → AudioClassifierBase<T> → SpeechEmotionRecognizer<T>

Implements: IEmotionRecognizer<T>

For Beginners

This is like teaching a computer to “hear” emotions in someone’s voice!

How it works:

  1. Audio is converted to a mel spectrogram (a visual representation of sound frequencies over time)
  2. A neural network analyzes patterns in the spectrogram
  3. The network outputs probabilities for each emotion (happy, sad, angry, etc.)

Key features detected:

  • Pitch patterns (high pitch often = excitement, low pitch often = sadness)
  • Speaking rate (fast = excited/angry, slow = sad/calm)
  • Volume dynamics (loud = angry, soft = sad/fearful)
  • Voice quality (breathy, tense, relaxed)

Common applications:

  • Call centers: Detect frustrated customers for priority handling
  • Mental health: Monitor patient emotional well-being
  • Voice assistants: Respond appropriately to user mood
  • Gaming: Adapt gameplay to player emotional state
  • Market research: Analyze focus group reactions

Default emotions supported (based on industry standards):

  • Neutral, Happy, Sad, Angry, Fearful, Disgusted, Surprised

You can also measure:

  • Arousal: How activated/calm the speaker is (-1 to +1)
  • Valence: How positive/negative the emotion is (-1 to +1)

How It Works

This model uses deep learning to detect emotions from speech audio. It supports two operation modes:

  • ONNX Mode: Load pre-trained models for fast inference
  • Native Mode: Train models from scratch with full customization

Example

var audioTensor = Tensor<float>.CreateRandom(1, 16000);
// Create a speech emotion recognizer using a pre-trained ONNX model
var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.AudioProcessing,
    inputSize: 80,
    outputSize: 7);
            
var recognizer = new SpeechEmotionRecognizer<float>(architecture: architecture, modelPath: "emotion_model.onnx", options: new SpeechEmotionRecognizerOptions { SampleRate = 16000 });
            
// Classify emotions from audio
var result = recognizer.RecognizeEmotion(audioTensor);
// result.Emotion: "Happy", "Sad", "Angry", etc.

Constructors

Constructor Summary
SpeechEmotionRecognizer(NeuralNetworkArchitecture<T> architecture, ILossFunction<T> lossFunction, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, SpeechEmotionRecognizerOptions options) Creates a speech emotion recognizer in native training mode.
SpeechEmotionRecognizer(NeuralNetworkArchitecture<T> architecture, string modelPath, SpeechEmotionRecognizerOptions options) Creates a speech emotion recognizer in ONNX inference mode with a pre-trained model.

Properties

Property Summary
IReadOnlyList<string> SupportedEmotions { get; }

Methods

Method Summary
EmotionResult<T> RecognizeEmotion(Tensor<T> audio)
IReadOnlyDictionary<string, T> GetEmotionProbabilities(Tensor<T> audio)
IReadOnlyList<TimedEmotionResult<T>> RecognizeEmotionTimeSeries(Tensor<T> audio, int windowSizeMs, int hopSizeMs)
ModelMetadata<T> GetModelMetadata()
T GetArousal(Tensor<T> audio)
T GetValence(Tensor<T> audio)
Vector<T> ExtractEmotionFeatures(Tensor<T> audio)