GlowTTS<T>

Models & Types · AiDotNet.TextToSpeech.Classic

Glow-TTS: a flow-based generative text-to-speech model that finds its own alignment with monotonic alignment search.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → GlowTTS<T>

Implements: IAcousticModel<T>

For Beginners

Glow-TTS learns an exactly reversible mapping between spectrograms and simple Gaussian noise shaped by the text; to speak, it draws noise shaped by the text and runs the mapping backwards.

How It Works

References: “Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search” (Kim et al., NeurIPS 2020) and its reference implementation (jaywalnut310/glow-tts) for what the paper leaves unstated.

The text encoder (§3.3, Fig. 7) embeds phonemes, refines them with a residual convolutional pre-net and a Transformer encoder with relative position representations (RelativePositionTransformerBlock), and projects each token to the mean of a Gaussian prior over mel frames (σ fixed at 1). The decoder (Fig. 8) is a normalizing flow over pairs of frames squeezed into channels: blocks of ActNormFlowLayer, GroupedInvertibleConvFlowLayer and AffineCouplingFlowLayer. Training maximizes the exact log-likelihood of the mel spectrogram under the prior aligned by monotonic alignment search (§3.2), plus a duration predictor’s log-domain MSE to the searched durations (on encoder outputs with their gradient stopped). Inference samples the expanded prior at temperature 0.333 and runs the flow in reverse.

The model aligns itself, so a plain Train(tokens, mel) is its full training signal.

Constructors

Constructor Summary
GlowTTS(NeuralNetworkArchitecture<T> architecture, GlowTTSOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
GlowTTS(NeuralNetworkArchitecture<T> architecture, string modelPath, GlowTTSOptions options)

Properties

Property Summary
int FftSize { get; }
int HopSize { get; }
int MaxTextLength { get; }
int MelChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> TextToMel(string text) Generates a mel spectrogram from text (the paper pairs it with WaveGlow).