PortaSpeech<T>
Models & Types · AiDotNet.TextToSpeech.Classic
PortaSpeech: a portable text-to-speech model with a mixture-alignment linguistic encoder, a VAE variational generator with a normalizing-flow prior, and a flow-based post-net with grouped parameter sharing.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → PortaSpeech<T>
Implements: IAcousticModel<T>
For Beginners
PortaSpeech plans speech word by word, sketches a spectrogram with a small variational autoencoder, and then sharpens the details with a normalizing flow, keeping the whole model small.
How It Works
References: “PortaSpeech: Portable and High-Quality Generative Text-to-Speech” (Ren et al., NeurIPS 2021) and its reference implementation (NATSpeech) for what the paper leaves unstated.
The linguistic encoder (§3.1, App. A.1) encodes phonemes with relative-position FFT layers, averages them per word
for a word encoder, predicts phoneme durations that sum to word durations, expands the word states over the frames
and attends from each frame to the phonemes of its own word, with learnable positional encodings scaled by the
relative position inside the word. The variational generator (§3.2, App. A.2) is a stride-4 VAE with WaveNet encoder
and decoder conditioned on those linguistic features, whose prior is a volume-preserving coupling flow; its KL is the
Monte-Carlo estimate of Eq. 3. The post-net (§3.3, App. A.3) is a conditional Glow whose WaveNets are shared within
groups of steps. Training (§3.4) minimizes the word duration loss, the reconstruction MAE and the KL, then the
post-net negative log-likelihood; see CurrentPhase.
Supervision: phoneme durations (Durations), summed per word. Words are
WordLengths when given, otherwise runs of non-space tokens with every space its
own boundary word.
Example
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.Regression,
inputSize: 200, outputSize: 80);
var model = new PortaSpeech<double>(architecture, new PortaSpeechOptions());
model.Train(new TtsTrainingSample<double> { Tokens = tokens, Mel = mel, Durations = durations });
model.CurrentPhase = PortaSpeechTrainingPhase.PostNet; // after the generator has converged
var mel = model.TextToMel("hello world");
Constructors
| Constructor | Summary |
|---|---|
PortaSpeech(NeuralNetworkArchitecture<T> architecture, PortaSpeechOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
PortaSpeech(NeuralNetworkArchitecture<T> architecture, string modelPath, PortaSpeechOptions options) |
Properties
| Property | Summary |
|---|---|
PortaSpeechTrainingPhase CurrentPhase { get; set; } |
The training stage (reference two-stage schedule): VariationalGenerator first, then PostNet (the reference switches after 160k updates). |
int FftSize { get; } |
|
int GeneratorUpdates { get; set; } |
Generator updates so far; the KL term is optimized from KlStartUpdates on. |
int HopSize { get; } |
|
int MaxTextLength { get; } |
|
int MelChannels { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> TextToMel(string text) |
Generates a mel spectrogram from text (the paper pairs it with HiFi-GAN). |