EmotiVoice<T>
Models & Types · AiDotNet.TextToSpeech.StyleEmotion
EmotiVoice: NetEase Youdao’s multi-voice, prompt-controlled emotional text-to-speech (2023).
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → EmotiVoice<T>
Implements: IEndToEndTts<T>
For Beginners
Tell EmotiVoice how to say something — “happy”, “sad”, “angry”, “whispering” — and pick one of its two thousand voices; it reads the text in that voice and mood.
How It Works
EmotiVoice has no paper; this is its released code (netease-youdao/EmotiVoice: models/prompt_tts_modified,
models/hifigan, train_am_vocoder_joint.py). An emotion prompt (“开心”, “very happy”) and the text itself
are embedded by a SimBERT style encoder (its pooled output); a PromptTTS-style acoustic model — an ESPnet Transformer
encoder over the phonemes, the speaker, prompt and content embeddings concatenated and projected, JETS’s alignment
module learning the durations, duration, pitch and energy predictors, Gaussian upsampling and a Transformer decoder —
produces an 80-dimensional latent per frame, and a HiFi-GAN V1 generator trained jointly with it (on 32-frame
segments) turns it into 16 kHz speech.
Constructors
| Constructor | Summary |
|---|---|
EmotiVoice(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
EmotiVoice(NeuralNetworkArchitecture<T> architecture, EmotiVoiceOptions options) |
Creates a native, trainable EmotiVoice. |
EmotiVoice(NeuralNetworkArchitecture<T> architecture, string modelPath, EmotiVoiceOptions options) |
Creates an ONNX-backed EmotiVoice for inference. |
Properties
| Property | Summary |
|---|---|
WordPieceTokenizer StyleTokenizer { get; } |
SimBERT’s tokenizer (its 13,685-token Chinese WordPiece vocabulary). |
bool TrainsOnSampleObjective { get; } |
|
int HiddenDim { get; } |
|
int MaxTextLength { get; } |
|
int NumFlowSteps { get; } |
|
int TrainingUpdates { get; set; } |
Training updates taken. |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
T TrainStyleEncoder(IReadOnlyList<int> promptIds, int pitch, int speed, int energy, int emotion) |
Pre-trains the style encoder on one prompt (the release’s style encoder training, simbert.py’s StylePretrainLoss): the cross-entropy of its pitch, speed, energy and emotion classifiers on the pooled SimBERT output. |
Tensor<T> StyleEmbedding(IReadOnlyList<int> simBertIds) |
SimBERT’s pooled output [1, 768] (tanh of a linear map of the [CLS] state) of token ids — the prompt’s or the content’s embedding (get_style_embedding), computed with the encoder frozen. |
Tensor<T> Synthesize(string text) |
|
TtsVoice<T> CreateVoice(int speaker, string prompt) |
A voice: a speaker of the released list by name (such as “8051”) and an emotion prompt. |
TtsVoice<T> CreateVoice(string speaker, string prompt) |
A voice: a speaker of the released list by name (such as “8051”) and an emotion prompt. |
int[] StyleTokens(string text) |
The SimBERT token ids ([CLS] … [SEP]) of a prompt or content text; an empty text is [CLS] [SEP], as Hugging Face’s tokenizer gives it. |
void LoadReferenceWeights(string path) |
Loads a released checkpoint flattened to named tensors: the acoustic model under am. and HiFi-GAN under generator. (the joint generator’s state dict, g_*), with or without a generator. or module. prefix. |
void LoadStyleEncoderWeights(string path) |
Loads the style encoder (SimBERT and the four prompt classifiers; the released style_encoder checkpoint’s model entry: bert.*, pitch_clf, speed_clf, energy_clf, emotion_clf), or a plain SimBERT checkpoint when it has no classi… |