VITS<T>
Models & Types · AiDotNet.TextToSpeech.EndToEnd
VITS: end-to-end text-to-speech as a conditional VAE whose prior is a normalizing flow over a text encoding, with a stochastic duration predictor and a HiFi-GAN decoder trained adversarially.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VitsTtsModelBase<T> → StochasticDurationVitsBase<T> → VITS<T>
For Beginners
VITS goes straight from text to a waveform in one network: it learns a hidden “voice space” from real audio, learns to predict that space from text, and learns a vocoder that turns it into sound.
How It Works
References: “Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech” (Kim et al., ICML 2021) and its reference implementation (jaywalnut310/vits) for what the paper leaves unstated.
The posterior encoder maps the linear spectrogram to z ~ q(z | x); the flow f_θ maps z to the prior space, where the text encoder’s per-token N(μ, σ) is aligned to the frames by monotonic alignment search (§2.2.1). The stochastic duration predictor learns a lower bound on the alignment’s durations (§2.2.2) and trains with everything else. A random 8192-sample window of z is decoded by the HiFi-GAN generator. The generator loss is the mel L1 × 45 + KL + duration + adversarial + feature matching (Eq. 9); the period and scale discriminators train on the LSGAN loss (Eq. 7). Synthesis samples durations (noise scale 0.8), expands the prior, draws z = μ + ε σ · 0.667, inverts the flow and decodes the whole sequence.
Constructors
| Constructor | Summary |
|---|---|
VITS(NeuralNetworkArchitecture<T> architecture, VITSOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a trainable VITS model. |
VITS(NeuralNetworkArchitecture<T> architecture, string modelPath, VITSOptions options) |
Creates a trainable VITS model. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |