WaveGlow<T>

Models & Types · AiDotNet.TextToSpeech.Vocoders

WaveGlow: a flow-based generative network for speech synthesis — Glow’s invertible 1×1 convolutions and affine couplings over groups of audio samples, with WaveNet-like coupling networks conditioned on the mel spectrogram, trained only by maximizing the likelihood.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VocoderBase<T> → SegmentVocoderBase<T> → WaveGlow<T>

For Beginners

WaveGlow learns a reversible mapping between speech and random noise; to make speech it draws noise and runs the mapping backwards, guided by the spectrogram.

How It Works

References: “WaveGlow: A Flow-based Generative Network for Speech Synthesis” (Prenger et al., ICASSP 2019) and NVIDIA/waveglow for what the paper leaves unstated.

Training maximizes log p(x) = −z(x)ᵀz(x) / 2σ² + Σ log s + Σ log|det W| (Eq. 12) of a clip, squeezed into groups of 8 samples and passed through the steps of flow with early outputs (§2.3). Synthesis draws z with a smaller σ (§2.4) and inverts every step.

Constructors

Constructor Summary
WaveGlow(NeuralNetworkArchitecture<T> architecture, WaveGlowOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a trainable WaveGlow.
WaveGlow(NeuralNetworkArchitecture<T> architecture, string modelPath, WaveGlowOptions options) Creates a trainable WaveGlow.

Properties

Property Summary
int UpsampleFactor { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> Decode(Tensor<T> mel, Tensor<T> z) The waveform [1, 1, samples] of a latent z [1, group, samples / group] (laid out as Encode returns it) given the mel spectrogram: the exact inverse of Encode.
ValueTuple<Tensor<T>, Tensor<T>> Encode(Tensor<T> mel, Tensor<T> audio) The latent z [1, group, samples / group] of a waveform [samples] given its mel spectrogram [1, mel, frames], and the log-determinant of the flow.