Emotion2Vec<T>

Models & Types · AiDotNet.Audio.Emotion

emotion2vec universal speech emotion recognition model.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → AudioClassifierBase<T> → Emotion2Vec<T>

Implements: IEmotionRecognizer<T>

For Beginners

emotion2vec detects emotions from speech. Feed it audio of someone talking and it tells you what emotion they’re expressing (happy, sad, angry, etc.) along with a confidence score. It can also measure arousal (calm vs excited) and valence (negative vs positive).

Usage:

How It Works

emotion2vec (Ma et al., 2023) is a universal speech emotion representation model using self-supervised pre-training on unlabeled speech followed by fine-tuning. It achieves state-of-the-art results across multiple SER benchmarks with a single model.

Constructors

Constructor Summary
Emotion2Vec(NeuralNetworkArchitecture<T> architecture, Emotion2VecOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates an emotion2vec model in native training mode.
Emotion2Vec(NeuralNetworkArchitecture<T> architecture, string modelPath, Emotion2VecOptions options) Creates an emotion2vec model in native training mode.

Properties

Property Summary
IReadOnlyList<string> SupportedEmotions { get; }

Methods

Method Summary
EmotionResult<T> RecognizeEmotion(Tensor<T> audio)
IReadOnlyDictionary<string, T> GetEmotionProbabilities(Tensor<T> audio)
IReadOnlyList<TimedEmotionResult<T>> RecognizeEmotionTimeSeries(Tensor<T> audio, int windowSizeMs, int hopSizeMs)
ModelMetadata<T> GetModelMetadata()
T GetArousal(Tensor<T> audio)
T GetValence(Tensor<T> audio)
Tensor<T> ForwardForTraining(Tensor<T> input) Runs the native stack for tape training and pools frame logits into one embedding.
Vector<T> ExtractEmotionFeatures(Tensor<T> audio)