YourTTS<T>

Models & Types · AiDotNet.TextToSpeech.EndToEnd

YourTTS: zero-shot multi-speaker text-to-speech and voice conversion — VITS conditioned on speaker embeddings from a pretrained speaker encoder, with language embeddings for multilingual training.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VitsTtsModelBase<T> → StochasticDurationVitsBase<T> → YourTTS<T>

For Beginners

YourTTS can speak in the voice of someone it has never heard before: give it a few seconds of their speech and it extracts a “voice fingerprint” that steers every part of the synthesis.

How It Works

References: “YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone” (Casanova et al., ICML 2022) and its implementation in Coqui TTS (TTS/tts/models/vits.py, the recipes/vctk/yourtts recipe) for what the paper leaves unstated.

The network is VITS’s (VITS) with YourTTS’s changes (§2): the speaker is a d-vector from the H/ASP speaker encoder (§3.1) — of the given reference recording, or of the utterance itself in training — and conditions the posterior encoder and the flow’s coupling layers through their WaveNets’ global conditioning, the stochastic duration predictor and the decoder through projections added to their inputs; with several languages, a 4-dimensional language embedding is concatenated to every character embedding and also conditions the duration predictor. The speaker encoder is pretrained and frozen (load it with LoadSpeakerEncoder). With UseSpeakerConsistencyLoss the generator also minimizes −α · cos(φ(g), φ(h)) between the speaker embeddings of the real and generated decoder windows (Eq. 1; with gradient through φ into the generated audio, as the erratum and Coqui ≥ 0.12 have it).

Constructors

Constructor Summary
YourTTS(NeuralNetworkArchitecture<T> architecture, YourTTSOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a trainable YourTTS model.
YourTTS(NeuralNetworkArchitecture<T> architecture, string modelPath, YourTTSOptions options) Creates a trainable YourTTS model.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> ComputeSpeakerEmbedding(Tensor<T> recording) The d-vector [512] of a recording [samples] at the model’s sample rate.
void LoadSpeakerEncoder(IReadOnlyDictionary<string, Tensor<T>> state) Loads the pretrained H/ASP speaker encoder from a Coqui ResNetSpeakerEncoder state dictionary (the model entry of model_se.pth.tar), keyed by its parameter names (e.g.