WaveNet<T>
Models & Types · AiDotNet.TextToSpeech.Vocoders
WaveNet: a generative model for raw audio — an autoregressive stack of dilated causal convolutions that predicts a softmax over μ-law-quantized samples, here locally conditioned on the mel spectrogram as a vocoder.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VocoderBase<T> → SegmentVocoderBase<T> → WaveNet<T>
For Beginners
WaveNet writes audio one sample at a time, each time choosing among 256 loudness levels with probabilities computed from everything it has written so far and the spectrogram.
How It Works
References: “WaveNet: A Generative Model for Raw Audio” (van den Oord et al., 2016) and r9y9/wavenet_vocoder for what the paper leaves unstated.
Training maximizes Σ_t log p(x_t | x_1..x_{t−1}, y) (Eq. 1, §2.5) with teacher forcing: the network reads the
one-hot μ-law classes of the previous samples and the upsampled mel y and predicts a softmax over the next sample’s
class. Synthesis samples one class at a time from that softmax, feeding each back as the next input; each causal
convolution keeps the inputs it needs, so a step costs one column per layer.
Constructors
| Constructor | Summary |
|---|---|
WaveNet(NeuralNetworkArchitecture<T> architecture, WaveNetOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a trainable WaveNet. |
WaveNet(NeuralNetworkArchitecture<T> architecture, string modelPath, WaveNetOptions options) |
Creates a trainable WaveNet. |
Properties
| Property | Summary |
|---|---|
int ReceptiveField { get; } |
The receptive field of the network in samples. |
int UpsampleFactor { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> LogProbabilities(Tensor<T> mel, Tensor<T> audio) |
The log-probabilities [1, classes, samples] of every sample’s class given the previous samples of audio``[samples] and the mel spectrogram [1, mel, frames] (teacher forcing; the sample before the first is silence). |
int[] Quantize(Tensor<T> audio) |
The μ-law classes of a waveform [samples]. |