Tacotron<T>

Models & Types · AiDotNet.TextToSpeech.Classic

Tacotron: an end-to-end attention-based sequence-to-sequence text-to-speech model with CBHG modules.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → Tacotron<T>

Implements: IAcousticModel<T>

For Beginners

Tacotron reads text and, step by step, paints the spectrogram while learning on its own which letters it is reading at each step; a second network sharpens the result into a full spectrogram that can be turned into sound.

How It Works

References: “Tacotron: Towards End-to-End Speech Synthesis” (Wang et al., Interspeech 2017).

The architecture is Table 1 of the paper: a character embedding, a pre-net and a CBHG encoder (CbhgLayer); a decoder pre-net feeding a 256-cell attention GRU with content-based tanh attention over the encoder outputs; the attention GRU output and context projected into two residual 256-cell decoder GRUs, which emit r = 2 mel frames per step from an all-zero frame; and a post-processing CBHG that predicts the linear-frequency spectrogram, turned into a waveform by Griffin–Lim. The loss is the L1 error of both spectrograms. Unstated details (padding, normalization order, the attention cell’s input) follow the reference implementation keithito/tacotron and are noted where they are applied.

Training data: the post-processing net trains on the linear spectrogram, which a mel spectrogram does not carry, so a plain Train(tokens, mel) throws; pass a TtsTrainingSample with the recording (Audio, from which both spectrograms are computed with the paper’s analysis settings) or with LinearSpectrogram.

Constructors

Constructor Summary
Tacotron(NeuralNetworkArchitecture<T> architecture, TacotronOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
Tacotron(NeuralNetworkArchitecture<T> architecture, string modelPath, TacotronOptions options)

Properties

Property Summary
int FftSize { get; }
int HopSize { get; }
int MaxTextLength { get; }
int MelChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> PredictLinearSpectrogram(string text) Predicts the log-magnitude linear spectrogram for text, the post-processing net’s output.
Tensor<T> SynthesizeWaveform(string text) Synthesizes a waveform the paper’s way (§3.4): Griffin–Lim (50 iterations) on the predicted magnitudes raised to the power 1.2, then de-emphasis.
Tensor<T> TextToMel(string text) Generates a log-mel spectrogram from text, the decoder’s targets.