Tacotron2<T>
Models & Types · AiDotNet.TextToSpeech.Classic
Tacotron 2’s spectrogram prediction network (Shen et al. 2018, §2.2): characters to a mel spectrogram through a convolutional and bidirectional-LSTM encoder, location-sensitive attention, and an autoregressive decoder of two zoneout LSTMs with a stop token and a residual post-net.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → Tacotron2<T>
Implements: IAcousticModel<T>
For Beginners
Tacotron 2 reads text one character at a time and writes a spectrogram one frame at a time, choosing at each step which characters to “look at”; a vocoder turns the spectrogram into audio.
How It Works
Encoder: a 512-wide character embedding, 3 convolutions of 512 filters (kernel 5) with batch normalization, ReLU and dropout 0.5, and a bidirectional LSTM of 256 units per direction. Decoder, per frame: a pre-net of 2 bias-free 256-unit ReLU layers whose dropout 0.5 stays on at inference, an attention LSTM of 1024 units, location-sensitive attention (128 wide, 32 location filters of length 31) over the previous and cumulative weights, a decoder LSTM of 1024 units, and linear projections of [decoder output; context] to the frame and the stop token. Both LSTMs use zoneout 0.1 (its expectation at inference). Post-net: 5 convolutions of 512 filters (kernel 5) with tanh but the last, added to the decoder’s frames.
Training (§3.1): teacher forcing, the summed MSE of the frames before and after the post-net plus the stop token’s binary cross-entropy; Adam (0.9, 0.999, ε = 1e-6) at 1e-3, L2 weight 1e-6.
Example
var tacotron = new Tacotron2<float>(architecture, new Tacotron2Options());
tacotron.Train(new TtsTrainingSample<float> { Tokens = characters, Audio = recording });
var mel = tacotron.TextToMel("Hello world.");
Constructors
| Constructor | Summary |
|---|---|
Tacotron2(NeuralNetworkArchitecture<T> architecture, Tacotron2Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a trainable Tacotron 2. |
Tacotron2(NeuralNetworkArchitecture<T> architecture, string modelPath, Tacotron2Options options) |
Creates a trainable Tacotron 2. |
Properties
| Property | Summary |
|---|---|
int FftSize { get; } |
|
int HopSize { get; } |
|
int MaxTextLength { get; } |
|
int MelChannels { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> TextToMel(string text) |