EmotiVoice<T>

Models & Types · AiDotNet.TextToSpeech.StyleEmotion

EmotiVoice: NetEase Youdao’s multi-voice, prompt-controlled emotional text-to-speech (2023).

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → EmotiVoice<T>

Implements: IEndToEndTts<T>

For Beginners

Tell EmotiVoice how to say something — “happy”, “sad”, “angry”, “whispering” — and pick one of its two thousand voices; it reads the text in that voice and mood.

How It Works

EmotiVoice has no paper; this is its released code (netease-youdao/EmotiVoice: models/prompt_tts_modified, models/hifigan, train_am_vocoder_joint.py). An emotion prompt (“开心”, “very happy”) and the text itself are embedded by a SimBERT style encoder (its pooled output); a PromptTTS-style acoustic model — an ESPnet Transformer encoder over the phonemes, the speaker, prompt and content embeddings concatenated and projected, JETS’s alignment module learning the durations, duration, pitch and energy predictors, Gaussian upsampling and a Transformer decoder — produces an 80-dimensional latent per frame, and a HiFi-GAN V1 generator trained jointly with it (on 32-frame segments) turns it into 16 kHz speech.

Constructors

Constructor Summary
EmotiVoice(NeuralNetworkArchitecture<T> architecture) Calls the fully specified overload with the omitted parameters’ defaults.
EmotiVoice(NeuralNetworkArchitecture<T> architecture, EmotiVoiceOptions options) Creates a native, trainable EmotiVoice.
EmotiVoice(NeuralNetworkArchitecture<T> architecture, string modelPath, EmotiVoiceOptions options) Creates an ONNX-backed EmotiVoice for inference.

Properties

Property Summary
WordPieceTokenizer StyleTokenizer { get; } SimBERT’s tokenizer (its 13,685-token Chinese WordPiece vocabulary).
bool TrainsOnSampleObjective { get; }
int HiddenDim { get; }
int MaxTextLength { get; }
int NumFlowSteps { get; }
int TrainingUpdates { get; set; } Training updates taken.

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
T TrainStyleEncoder(IReadOnlyList<int> promptIds, int pitch, int speed, int energy, int emotion) Pre-trains the style encoder on one prompt (the release’s style encoder training, simbert.py’s StylePretrainLoss): the cross-entropy of its pitch, speed, energy and emotion classifiers on the pooled SimBERT output.
Tensor<T> StyleEmbedding(IReadOnlyList<int> simBertIds) SimBERT’s pooled output [1, 768] (tanh of a linear map of the [CLS] state) of token ids — the prompt’s or the content’s embedding (get_style_embedding), computed with the encoder frozen.
Tensor<T> Synthesize(string text)
TtsVoice<T> CreateVoice(int speaker, string prompt) A voice: a speaker of the released list by name (such as “8051”) and an emotion prompt.
TtsVoice<T> CreateVoice(string speaker, string prompt) A voice: a speaker of the released list by name (such as “8051”) and an emotion prompt.
int[] StyleTokens(string text) The SimBERT token ids ([CLS] … [SEP]) of a prompt or content text; an empty text is [CLS] [SEP], as Hugging Face’s tokenizer gives it.
void LoadReferenceWeights(string path) Loads a released checkpoint flattened to named tensors: the acoustic model under am. and HiFi-GAN under generator. (the joint generator’s state dict, g_*), with or without a generator. or module. prefix.
void LoadStyleEncoderWeights(string path) Loads the style encoder (SimBERT and the four prompt classifiers; the released style_encoder checkpoint’s model entry: bert.*, pitch_clf, speed_clf, energy_clf, emotion_clf), or a plain SimBERT checkpoint when it has no classi…