VarianceAdaptorTtsModelBase<T>

Base Classes · AiDotNet.TextToSpeech

Base class for non-autoregressive acoustic models of the FastSpeech family: a phoneme encoder, a variance adaptor (duration and length regulation, optionally pitch and energy), and a mel decoder.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VarianceAdaptorTtsModelBase<T>

Known derived types: AdaSpeech<T>, FastSpeech2<T>, FastSpeech<T>, ProDiff<T>, PromptTTS<T>

How It Works

The layer stack is encoder layers, , decoder layers, registered with AddEncoderDecoderLayers so the adaptor sits at Layers[EncoderLayerCount]. Inference is an ordinary sequential forward (the adaptor uses its predictions). Training is the papers’ teacher-forced objective: forced-alignment durations, and pitch and energy when the adaptor has those branches, drive the adaptor, and the loss is the mel reconstruction error plus the mean squared error of each predictor (FastSpeech, Ren et al. 2019 §3.3; FastSpeech 2, Ren et al. 2021 §2.3).

A model supplies its layers and its paper’s mel loss; the training and inference plumbing is shared.

Properties

Property Summary
VarianceAdaptorLayer<T> VarianceAdaptor { get; } The variance adaptor between the phoneme encoder and the mel decoder.

Methods

Method Summary
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
void Train(Tensor<T> input, Tensor<T> expectedOutput)