WaveNet<T>

Models & Types · AiDotNet.TextToSpeech.Vocoders

WaveNet: a generative model for raw audio — an autoregressive stack of dilated causal convolutions that predicts a softmax over μ-law-quantized samples, here locally conditioned on the mel spectrogram as a vocoder.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VocoderBase<T> → SegmentVocoderBase<T> → WaveNet<T>

For Beginners

WaveNet writes audio one sample at a time, each time choosing among 256 loudness levels with probabilities computed from everything it has written so far and the spectrogram.

How It Works

References: “WaveNet: A Generative Model for Raw Audio” (van den Oord et al., 2016) and r9y9/wavenet_vocoder for what the paper leaves unstated.

Training maximizes Σ_t log p(x_t | x_1..x_{t−1}, y) (Eq. 1, §2.5) with teacher forcing: the network reads the one-hot μ-law classes of the previous samples and the upsampled mel y and predicts a softmax over the next sample’s class. Synthesis samples one class at a time from that softmax, feeding each back as the next input; each causal convolution keeps the inputs it needs, so a step costs one column per layer.

Constructors

Constructor Summary
WaveNet(NeuralNetworkArchitecture<T> architecture, WaveNetOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a trainable WaveNet.
WaveNet(NeuralNetworkArchitecture<T> architecture, string modelPath, WaveNetOptions options) Creates a trainable WaveNet.

Properties

Property Summary
int ReceptiveField { get; } The receptive field of the network in samples.
int UpsampleFactor { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> LogProbabilities(Tensor<T> mel, Tensor<T> audio) The log-probabilities [1, classes, samples] of every sample’s class given the previous samples of audio``[samples] and the mel spectrogram [1, mel, frames] (teacher forcing; the sample before the first is silence).
int[] Quantize(Tensor<T> audio) The μ-law classes of a waveform [samples].