VitsTtsModelBase<T>

Base Classes · AiDotNet.TextToSpeech.EndToEnd

The VITS family’s shared model: a conditional VAE whose prior is a normalizing flow over a text encoding, aligned by monotonic alignment search, with a duration model and a HiFi-GAN decoder trained adversarially on random windows.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VitsTtsModelBase<T>

Implements: IEndToEndTts<T>

Known derived types: StochasticDurationVitsBase<T>, VITS2<T>

For Beginners

A VITS-style model goes straight from text to a waveform in one network: it learns a hidden “voice space” from real audio, learns to predict that space from text, and learns a vocoder that turns it into sound.

How It Works

Everything VITS (Kim et al. 2021) defines and its successors keep lives here: the relative-position text encoder and its prior projection; the posterior encoder q(z | x) (WaveNet over the linear spectrogram, or over the mel spectrogram when PosteriorReadsMel); the residual-coupling flow (optionally with VITS2’s Transformer block); monotonic alignment search on log N(f(z); μ, σ) (optionally with VITS2’s noise); the KL term through the flow; the HiFi-GAN decoder on a random window; the period and scale discriminators with the LSGAN and feature-matching losses; a speaker table conditioning the posterior, the flow, the decoder and the duration model when there are several speakers; AdamW with the per-epoch decay.

A model supplies its duration model: how it is built (CreateDurationModel), what it adds to the generator loss (JointDurationLoss), and how it predicts durations for synthesis (PredictDurations). A model whose paper trains more than the joint generator/discriminator step overrides TrainUtterance and uses TrainLayers and Align.

Properties

Property Summary
int HiddenDim { get; } Gets the text encoder’s width.
int MaxTextLength { get; } Gets the longest text the model accepts.
int NumFlowSteps { get; } Gets the number of coupling layers in the flow.

Methods

Method Summary
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
void Train(Tensor<T> input, Tensor<T> expectedOutput)