PortaSpeech<T>

Models & Types · AiDotNet.TextToSpeech.Classic

PortaSpeech: a portable text-to-speech model with a mixture-alignment linguistic encoder, a VAE variational generator with a normalizing-flow prior, and a flow-based post-net with grouped parameter sharing.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → PortaSpeech<T>

Implements: IAcousticModel<T>

For Beginners

PortaSpeech plans speech word by word, sketches a spectrogram with a small variational autoencoder, and then sharpens the details with a normalizing flow, keeping the whole model small.

How It Works

References: “PortaSpeech: Portable and High-Quality Generative Text-to-Speech” (Ren et al., NeurIPS 2021) and its reference implementation (NATSpeech) for what the paper leaves unstated.

The linguistic encoder (§3.1, App. A.1) encodes phonemes with relative-position FFT layers, averages them per word for a word encoder, predicts phoneme durations that sum to word durations, expands the word states over the frames and attends from each frame to the phonemes of its own word, with learnable positional encodings scaled by the relative position inside the word. The variational generator (§3.2, App. A.2) is a stride-4 VAE with WaveNet encoder and decoder conditioned on those linguistic features, whose prior is a volume-preserving coupling flow; its KL is the Monte-Carlo estimate of Eq. 3. The post-net (§3.3, App. A.3) is a conditional Glow whose WaveNets are shared within groups of steps. Training (§3.4) minimizes the word duration loss, the reconstruction MAE and the KL, then the post-net negative log-likelihood; see CurrentPhase.

Supervision: phoneme durations (Durations), summed per word. Words are WordLengths when given, otherwise runs of non-space tokens with every space its own boundary word.

Example

var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.Regression,
    inputSize: 200, outputSize: 80);
var model = new PortaSpeech<double>(architecture, new PortaSpeechOptions());
model.Train(new TtsTrainingSample<double> { Tokens = tokens, Mel = mel, Durations = durations });
model.CurrentPhase = PortaSpeechTrainingPhase.PostNet;   // after the generator has converged
var mel = model.TextToMel("hello world");

Constructors

Constructor Summary
PortaSpeech(NeuralNetworkArchitecture<T> architecture, PortaSpeechOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
PortaSpeech(NeuralNetworkArchitecture<T> architecture, string modelPath, PortaSpeechOptions options)

Properties

Property Summary
PortaSpeechTrainingPhase CurrentPhase { get; set; } The training stage (reference two-stage schedule): VariationalGenerator first, then PostNet (the reference switches after 160k updates).
int FftSize { get; }
int GeneratorUpdates { get; set; } Generator updates so far; the KL term is optimized from KlStartUpdates on.
int HopSize { get; }
int MaxTextLength { get; }
int MelChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> TextToMel(string text) Generates a mel spectrogram from text (the paper pairs it with HiFi-GAN).