VITS<T>

Models & Types · AiDotNet.TextToSpeech.EndToEnd

VITS: end-to-end text-to-speech as a conditional VAE whose prior is a normalizing flow over a text encoding, with a stochastic duration predictor and a HiFi-GAN decoder trained adversarially.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VitsTtsModelBase<T> → StochasticDurationVitsBase<T> → VITS<T>

For Beginners

VITS goes straight from text to a waveform in one network: it learns a hidden “voice space” from real audio, learns to predict that space from text, and learns a vocoder that turns it into sound.

How It Works

References: “Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech” (Kim et al., ICML 2021) and its reference implementation (jaywalnut310/vits) for what the paper leaves unstated.

The posterior encoder maps the linear spectrogram to z ~ q(z | x); the flow f_θ maps z to the prior space, where the text encoder’s per-token N(μ, σ) is aligned to the frames by monotonic alignment search (§2.2.1). The stochastic duration predictor learns a lower bound on the alignment’s durations (§2.2.2) and trains with everything else. A random 8192-sample window of z is decoded by the HiFi-GAN generator. The generator loss is the mel L1 × 45 + KL + duration + adversarial + feature matching (Eq. 9); the period and scale discriminators train on the LSGAN loss (Eq. 7). Synthesis samples durations (noise scale 0.8), expands the prior, draws z = μ + ε σ · 0.667, inverts the flow and decodes the whole sequence.

Constructors

Constructor Summary
VITS(NeuralNetworkArchitecture<T> architecture, VITSOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a trainable VITS model.
VITS(NeuralNetworkArchitecture<T> architecture, string modelPath, VITSOptions options) Creates a trainable VITS model.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()