YourTTS<T>
Models & Types · AiDotNet.TextToSpeech.EndToEnd
YourTTS: zero-shot multi-speaker text-to-speech and voice conversion — VITS conditioned on speaker embeddings from a pretrained speaker encoder, with language embeddings for multilingual training.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VitsTtsModelBase<T> → StochasticDurationVitsBase<T> → YourTTS<T>
For Beginners
YourTTS can speak in the voice of someone it has never heard before: give it a few seconds of their speech and it extracts a “voice fingerprint” that steers every part of the synthesis.
How It Works
References: “YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone”
(Casanova et al., ICML 2022) and its implementation in Coqui TTS (TTS/tts/models/vits.py, the
recipes/vctk/yourtts recipe) for what the paper leaves unstated.
The network is VITS’s (VITS) with YourTTS’s changes (§2): the speaker is a d-vector from the H/ASP
speaker encoder (§3.1) — of the given reference recording, or of the utterance itself in training — and conditions the
posterior encoder and the flow’s coupling layers through their WaveNets’ global conditioning, the stochastic duration
predictor and the decoder through projections added to their inputs; with several languages, a 4-dimensional language
embedding is concatenated to every character embedding and also conditions the duration predictor. The speaker
encoder is pretrained and frozen (load it with LoadSpeakerEncoder). With
UseSpeakerConsistencyLoss the generator also minimizes
−α · cos(φ(g), φ(h)) between the speaker embeddings of the real and generated decoder windows (Eq. 1; with
gradient through φ into the generated audio, as the erratum and Coqui ≥ 0.12 have it).
Constructors
| Constructor | Summary |
|---|---|
YourTTS(NeuralNetworkArchitecture<T> architecture, YourTTSOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a trainable YourTTS model. |
YourTTS(NeuralNetworkArchitecture<T> architecture, string modelPath, YourTTSOptions options) |
Creates a trainable YourTTS model. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> ComputeSpeakerEmbedding(Tensor<T> recording) |
The d-vector [512] of a recording [samples] at the model’s sample rate. |
void LoadSpeakerEncoder(IReadOnlyDictionary<string, Tensor<T>> state) |
Loads the pretrained H/ASP speaker encoder from a Coqui ResNetSpeakerEncoder state dictionary (the model entry of model_se.pth.tar), keyed by its parameter names (e.g. |