Tacotron<T>
Models & Types · AiDotNet.TextToSpeech.Classic
Tacotron: an end-to-end attention-based sequence-to-sequence text-to-speech model with CBHG modules.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → Tacotron<T>
Implements: IAcousticModel<T>
For Beginners
Tacotron reads text and, step by step, paints the spectrogram while learning on its own which letters it is reading at each step; a second network sharpens the result into a full spectrogram that can be turned into sound.
How It Works
References: “Tacotron: Towards End-to-End Speech Synthesis” (Wang et al., Interspeech 2017).
The architecture is Table 1 of the paper: a character embedding, a pre-net and a CBHG encoder
(CbhgLayer); a decoder pre-net feeding a 256-cell attention GRU with content-based tanh attention
over the encoder outputs; the attention GRU output and context projected into two residual 256-cell decoder GRUs,
which emit r = 2 mel frames per step from an all-zero
Training data: the post-processing net trains on the linear spectrogram, which a mel spectrogram does not carry, so
a plain Train(tokens, mel) throws; pass a TtsTrainingSample with the recording
(Audio, from which both spectrograms are computed with the paper’s analysis
settings) or with LinearSpectrogram.
Constructors
| Constructor | Summary |
|---|---|
Tacotron(NeuralNetworkArchitecture<T> architecture, TacotronOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
Tacotron(NeuralNetworkArchitecture<T> architecture, string modelPath, TacotronOptions options) |
Properties
| Property | Summary |
|---|---|
int FftSize { get; } |
|
int HopSize { get; } |
|
int MaxTextLength { get; } |
|
int MelChannels { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> PredictLinearSpectrogram(string text) |
Predicts the log-magnitude linear spectrogram for text, the post-processing net’s output. |
Tensor<T> SynthesizeWaveform(string text) |
Synthesizes a waveform the paper’s way (§3.4): Griffin–Lim (50 iterations) on the predicted magnitudes raised to the power 1.2, then de-emphasis. |
Tensor<T> TextToMel(string text) |
Generates a log-mel spectrogram from text, the decoder’s targets. |