WaveRNN<T>

Models & Types · AiDotNet.TextToSpeech.Vocoders

WaveRNN: efficient neural audio synthesis with a single recurrent layer whose state is split to predict the coarse and the fine byte of each 16-bit sample through a dual softmax.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VocoderBase<T> → SegmentVocoderBase<T> → WaveRNN<T>

For Beginners

WaveRNN writes audio one sample at a time with a small recurrent network: it first chooses the sample’s rough level (the top 8 bits), then refines it (the bottom 8 bits).

How It Works

References: “Efficient Neural Audio Synthesis” (Kalchbrenner et al., ICML 2018).

Training maximizes log P(c_t) + log P(f_t | c_t) of every sample of a 960-sample sequence with full back-propagation through time (§2, §5), the inputs being the true previous sample and current coarse byte. Synthesis runs the cell once per sample: the coarse half of the state does not see the current coarse byte, so the coarse byte is drawn first and the fine byte second from the state that has seen it. Magnitude pruning of the recurrent weights (§3, Sparse WaveRNN) runs when a sparsity target is set.

Constructors

Constructor Summary
WaveRNN(NeuralNetworkArchitecture<T> architecture, WaveRNNOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a trainable WaveRNN.
WaveRNN(NeuralNetworkArchitecture<T> architecture, string modelPath, WaveRNNOptions options) Creates a trainable WaveRNN.

Properties

Property Summary
double RecurrentSparsity { get; } The fraction of the recurrent weights that are zero.
int UpsampleFactor { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
ValueTuple<Tensor<T>, Tensor<T>> Logits(Tensor<T> mel, Tensor<T> audio) The coarse and fine logits [1, 256, samples] of every sample of audio given its previous samples and coarse byte (teacher forcing; the sample before the first is silence).
ValueTuple<int[], int[]> Split(Tensor<T> audio) The coarse (high) and fine (low) bytes of a waveform [samples].