VITS2<T>

Models & Types · AiDotNet.TextToSpeech.EndToEnd

VITS2: single-stage text-to-speech that improves VITS with an adversarially trained stochastic duration predictor, noise-scaled monotonic alignment search, a Transformer block in the normalizing flows and a speaker-conditioned text encoder.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VitsTtsModelBase<T> → VITS2<T>

For Beginners

VITS2 goes straight from text to a waveform like VITS, but learns how long each sound lasts by competing against a judge that tells real durations from predicted ones, which makes speech sound more natural.

How It Works

References: “VITS2: Improving Quality and Efficiency of Single-Stage Text-to-Speech with Adversarial Learning and Architecture Design” (Kong et al., Interspeech 2023). The paper has no official code; what it leaves open follows the community reference implementation (p0p4k/vits2_pytorch), as listed on VITS2Options.

Training first fits the waveform networks as VITS does, without its duration model: the reconstruction, KL, adversarial and feature-matching losses, with Gaussian noise ε = std(P) · N(0, 1) · s on the alignment scores while s = 0.01 − 2·10⁻⁶ · step is positive (§2.2), the mel spectrogram as the posterior encoder’s input and a Transformer block in every flow coupling (§2.3). After AcousticTrainingSteps steps the duration predictor G(z_d, h_text) trains on its own (§2.1, “separately trained as the last training step”) against the alignment’s log-durations d with L_mse (Eq. 3) + L_adv(G) (Eq. 2), and the time-step-wise discriminator D(d, h_text) with L_adv(D) (Eq. 1). With several speakers the speaker embedding also enters the text encoder before its third block (§2.4). Synthesis draws z_d, predicts log-durations, expands the prior, samples it and inverts the flow.

Constructors

Constructor Summary
VITS2(NeuralNetworkArchitecture<T> architecture, VITS2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a trainable VITS2 model.
VITS2(NeuralNetworkArchitecture<T> architecture, string modelPath, VITS2Options options) Creates a trainable VITS2 model.

Properties

Property Summary
bool TrainsDurationPredictor { get; } Whether the next training step trains the duration predictor (after the waveform networks).

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()