Tacotron2<T>

Models & Types · AiDotNet.TextToSpeech.Classic

Tacotron 2’s spectrogram prediction network (Shen et al. 2018, §2.2): characters to a mel spectrogram through a convolutional and bidirectional-LSTM encoder, location-sensitive attention, and an autoregressive decoder of two zoneout LSTMs with a stop token and a residual post-net.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → Tacotron2<T>

Implements: IAcousticModel<T>

For Beginners

Tacotron 2 reads text one character at a time and writes a spectrogram one frame at a time, choosing at each step which characters to “look at”; a vocoder turns the spectrogram into audio.

How It Works

Encoder: a 512-wide character embedding, 3 convolutions of 512 filters (kernel 5) with batch normalization, ReLU and dropout 0.5, and a bidirectional LSTM of 256 units per direction. Decoder, per frame: a pre-net of 2 bias-free 256-unit ReLU layers whose dropout 0.5 stays on at inference, an attention LSTM of 1024 units, location-sensitive attention (128 wide, 32 location filters of length 31) over the previous and cumulative weights, a decoder LSTM of 1024 units, and linear projections of [decoder output; context] to the frame and the stop token. Both LSTMs use zoneout 0.1 (its expectation at inference). Post-net: 5 convolutions of 512 filters (kernel 5) with tanh but the last, added to the decoder’s frames.

Training (§3.1): teacher forcing, the summed MSE of the frames before and after the post-net plus the stop token’s binary cross-entropy; Adam (0.9, 0.999, ε = 1e-6) at 1e-3, L2 weight 1e-6.

Example

var tacotron = new Tacotron2<float>(architecture, new Tacotron2Options());
tacotron.Train(new TtsTrainingSample<float> { Tokens = characters, Audio = recording });
var mel = tacotron.TextToMel("Hello world.");

Constructors

Constructor Summary
Tacotron2(NeuralNetworkArchitecture<T> architecture, Tacotron2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a trainable Tacotron 2.
Tacotron2(NeuralNetworkArchitecture<T> architecture, string modelPath, Tacotron2Options options) Creates a trainable Tacotron 2.

Properties

Property Summary
int FftSize { get; }
int HopSize { get; }
int MaxTextLength { get; }
int MelChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> TextToMel(string text)