VitsTtsModelBase<T>
Base Classes · AiDotNet.TextToSpeech.EndToEnd
The VITS family’s shared model: a conditional VAE whose prior is a normalizing flow over a text encoding, aligned by monotonic alignment search, with a duration model and a HiFi-GAN decoder trained adversarially on random windows.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VitsTtsModelBase<T>
Implements: IEndToEndTts<T>
Known derived types: StochasticDurationVitsBase<T>, VITS2<T>
For Beginners
A VITS-style model goes straight from text to a waveform in one network: it learns a hidden “voice space” from real audio, learns to predict that space from text, and learns a vocoder that turns it into sound.
How It Works
Everything VITS (Kim et al. 2021) defines and its successors keep lives here: the relative-position text encoder and
its prior projection; the posterior encoder q(z | x) (WaveNet over the linear spectrogram, or over the mel spectrogram
when PosteriorReadsMel); the residual-coupling flow (optionally with VITS2’s Transformer block);
monotonic alignment search on log N(f(z); μ, σ) (optionally with VITS2’s noise); the KL term through the flow; the
HiFi-GAN decoder on a random window; the period and scale discriminators with the LSGAN and feature-matching losses;
a speaker table conditioning the posterior, the flow, the decoder and the duration model when there are several
speakers; AdamW with the per-epoch decay.
A model supplies its duration model: how it is built (CreateDurationModel), what it adds to the
generator loss (JointDurationLoss), and how it predicts durations for synthesis
(PredictDurations). A model whose paper trains more than the joint generator/discriminator step
overrides TrainUtterance and uses TrainLayers and Align.
Properties
| Property | Summary |
|---|---|
int HiddenDim { get; } |
Gets the text encoder’s width. |
int MaxTextLength { get; } |
Gets the longest text the model accepts. |
int NumFlowSteps { get; } |
Gets the number of coupling layers in the flow. |
Methods
| Method | Summary |
|---|---|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
void Train(Tensor<T> input, Tensor<T> expectedOutput) |