TransformerTTS<T>
Models & Types · AiDotNet.TextToSpeech.Classic
Transformer TTS: Tacotron 2’s text-to-mel pipeline with its recurrent encoder and decoder replaced by a Transformer.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → TransformerTTS<T>
Implements: IAcousticModel<T>
For Beginners
Like Tacotron 2, but every recurrent network is replaced by attention, so training runs over all frames in parallel.
How It Works
References: “Neural Speech Synthesis with Transformer Network” (Li et al., AAAI 2019).
The structure is the paper’s Fig. 3: phoneme embeddings through Tacotron 2’s 3-layer convolutional encoder pre-net and
a linear projection (§3.3), plus a scaled positional encoding with a trainable weight α (§3.2,
ScaledPositionalEncodingLayer); a Transformer encoder (§3.5); the mel spectrogram through a
2-layer fully connected decoder pre-net and a linear projection plus its own scaled positional encoding (§3.4); a
Transformer decoder with masked self-attention and encoder–decoder attention (§3.6,
PostNormTransformerDecoderBlock); mel and stop linear projections and a 5-layer convolutional
post-net producing a residual (§3.7). Training is teacher-forced and minimizes the mel mean squared error before and
after the post-net plus the stop-token binary cross-entropy with a positive weight on the final frame.
Inference is autoregressive until the stop token fires.
The model learns its own alignment through attention, so a plain Train(tokens, mel) is its full
training signal.
Constructors
| Constructor | Summary |
|---|---|
TransformerTTS(NeuralNetworkArchitecture<T> architecture, TransformerTTSOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
TransformerTTS(NeuralNetworkArchitecture<T> architecture, string modelPath, TransformerTTSOptions options) |
Properties
| Property | Summary |
|---|---|
T DecoderPositionScale { get; } |
The trained scale α of the decoder’s positional encoding. |
T EncoderPositionScale { get; } |
The trained scale α of the encoder’s positional encoding (§3.2, §4.6). |
int FftSize { get; } |
|
int HopSize { get; } |
|
int MaxTextLength { get; } |
|
int MelChannels { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> TextToMel(string text) |
Generates a mel spectrogram from text (the paper pairs it with a WaveNet vocoder). |