VITS2<T>
Models & Types · AiDotNet.TextToSpeech.EndToEnd
VITS2: single-stage text-to-speech that improves VITS with an adversarially trained stochastic duration predictor, noise-scaled monotonic alignment search, a Transformer block in the normalizing flows and a speaker-conditioned text encoder.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VitsTtsModelBase<T> → VITS2<T>
For Beginners
VITS2 goes straight from text to a waveform like VITS, but learns how long each sound lasts by competing against a judge that tells real durations from predicted ones, which makes speech sound more natural.
How It Works
References: “VITS2: Improving Quality and Efficiency of Single-Stage Text-to-Speech with Adversarial
Learning and Architecture Design” (Kong et al., Interspeech 2023). The paper has no official code; what it leaves open
follows the community reference implementation (p0p4k/vits2_pytorch), as listed on VITS2Options.
Training first fits the waveform networks as VITS does, without its duration model: the reconstruction, KL,
adversarial and feature-matching losses, with Gaussian noise ε = std(P) · N(0, 1) · s on the alignment scores while
s = 0.01 − 2·10⁻⁶ · step is positive (§2.2), the mel spectrogram as the posterior encoder’s input and a Transformer
block in every flow coupling (§2.3). After AcousticTrainingSteps steps the duration
predictor G(z_d, h_text) trains on its own (§2.1, “separately trained as the last training step”) against the
alignment’s log-durations d with L_mse (Eq. 3) + L_adv(G) (Eq. 2), and the time-step-wise discriminator
D(d, h_text) with L_adv(D) (Eq. 1). With several speakers the speaker embedding also enters the text encoder before its
third block (§2.4). Synthesis draws z_d, predicts log-durations, expands the prior, samples it and inverts the flow.
Constructors
| Constructor | Summary |
|---|---|
VITS2(NeuralNetworkArchitecture<T> architecture, VITS2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a trainable VITS2 model. |
VITS2(NeuralNetworkArchitecture<T> architecture, string modelPath, VITS2Options options) |
Creates a trainable VITS2 model. |
Properties
| Property | Summary |
|---|---|
bool TrainsDurationPredictor { get; } |
Whether the next training step trains the duration predictor (after the waveform networks). |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |