SpeechEmotionRecognizer<T>
Models & Types · AiDotNet.Audio.Emotion
Neural network-based speech emotion recognition model that classifies emotional states from audio.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → AudioClassifierBase<T> → SpeechEmotionRecognizer<T>
Implements: IEmotionRecognizer<T>
For Beginners
This is like teaching a computer to “hear” emotions in someone’s voice!
How it works:
- Audio is converted to a mel spectrogram (a visual representation of sound frequencies over time)
- A neural network analyzes patterns in the spectrogram
- The network outputs probabilities for each emotion (happy, sad, angry, etc.)
Key features detected:
- Pitch patterns (high pitch often = excitement, low pitch often = sadness)
- Speaking rate (fast = excited/angry, slow = sad/calm)
- Volume dynamics (loud = angry, soft = sad/fearful)
- Voice quality (breathy, tense, relaxed)
Common applications:
- Call centers: Detect frustrated customers for priority handling
- Mental health: Monitor patient emotional well-being
- Voice assistants: Respond appropriately to user mood
- Gaming: Adapt gameplay to player emotional state
- Market research: Analyze focus group reactions
Default emotions supported (based on industry standards):
- Neutral, Happy, Sad, Angry, Fearful, Disgusted, Surprised
You can also measure:
- Arousal: How activated/calm the speaker is (-1 to +1)
- Valence: How positive/negative the emotion is (-1 to +1)
How It Works
This model uses deep learning to detect emotions from speech audio. It supports two operation modes:
- ONNX Mode: Load pre-trained models for fast inference
- Native Mode: Train models from scratch with full customization
Example
var audioTensor = Tensor<float>.CreateRandom(1, 16000);
// Create a speech emotion recognizer using a pre-trained ONNX model
var architecture = new NeuralNetworkArchitecture<float>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.AudioProcessing,
inputSize: 80,
outputSize: 7);
var recognizer = new SpeechEmotionRecognizer<float>(architecture: architecture, modelPath: "emotion_model.onnx", options: new SpeechEmotionRecognizerOptions { SampleRate = 16000 });
// Classify emotions from audio
var result = recognizer.RecognizeEmotion(audioTensor);
// result.Emotion: "Happy", "Sad", "Angry", etc.
Constructors
| Constructor | Summary |
|---|---|
SpeechEmotionRecognizer(NeuralNetworkArchitecture<T> architecture, ILossFunction<T> lossFunction, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, SpeechEmotionRecognizerOptions options) |
Creates a speech emotion recognizer in native training mode. |
SpeechEmotionRecognizer(NeuralNetworkArchitecture<T> architecture, string modelPath, SpeechEmotionRecognizerOptions options) |
Creates a speech emotion recognizer in ONNX inference mode with a pre-trained model. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedEmotions { get; } |
Methods
| Method | Summary |
|---|---|
EmotionResult<T> RecognizeEmotion(Tensor<T> audio) |
|
IReadOnlyDictionary<string, T> GetEmotionProbabilities(Tensor<T> audio) |
|
IReadOnlyList<TimedEmotionResult<T>> RecognizeEmotionTimeSeries(Tensor<T> audio, int windowSizeMs, int hopSizeMs) |
|
ModelMetadata<T> GetModelMetadata() |
|
T GetArousal(Tensor<T> audio) |
|
T GetValence(Tensor<T> audio) |
|
Vector<T> ExtractEmotionFeatures(Tensor<T> audio) |