WaveRNN<T>
Models & Types · AiDotNet.TextToSpeech.Vocoders
WaveRNN: efficient neural audio synthesis with a single recurrent layer whose state is split to predict the coarse and the fine byte of each 16-bit sample through a dual softmax.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VocoderBase<T> → SegmentVocoderBase<T> → WaveRNN<T>
For Beginners
WaveRNN writes audio one sample at a time with a small recurrent network: it first chooses the sample’s rough level (the top 8 bits), then refines it (the bottom 8 bits).
How It Works
References: “Efficient Neural Audio Synthesis” (Kalchbrenner et al., ICML 2018).
Training maximizes log P(c_t) + log P(f_t | c_t) of every sample of a 960-sample sequence with full
back-propagation through time (§2, §5), the inputs being the true previous sample and current coarse byte. Synthesis
runs the cell once per sample: the coarse half of the state does not see the current coarse byte, so the coarse byte
is drawn first and the fine byte second from the state that has seen it. Magnitude pruning of the recurrent weights
(§3, Sparse WaveRNN) runs when a sparsity target is set.
Constructors
| Constructor | Summary |
|---|---|
WaveRNN(NeuralNetworkArchitecture<T> architecture, WaveRNNOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a trainable WaveRNN. |
WaveRNN(NeuralNetworkArchitecture<T> architecture, string modelPath, WaveRNNOptions options) |
Creates a trainable WaveRNN. |
Properties
| Property | Summary |
|---|---|
double RecurrentSparsity { get; } |
The fraction of the recurrent weights that are zero. |
int UpsampleFactor { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
ValueTuple<Tensor<T>, Tensor<T>> Logits(Tensor<T> mel, Tensor<T> audio) |
The coarse and fine logits [1, 256, samples] of every sample of audio given its previous samples and coarse byte (teacher forcing; the sample before the first is silence). |
ValueTuple<int[], int[]> Split(Tensor<T> audio) |
The coarse (high) and fine (low) bytes of a waveform [samples]. |