TransformerTTS<T>

Models & Types · AiDotNet.TextToSpeech.Classic

Transformer TTS: Tacotron 2’s text-to-mel pipeline with its recurrent encoder and decoder replaced by a Transformer.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → TransformerTTS<T>

Implements: IAcousticModel<T>

For Beginners

Like Tacotron 2, but every recurrent network is replaced by attention, so training runs over all frames in parallel.

How It Works

References: “Neural Speech Synthesis with Transformer Network” (Li et al., AAAI 2019).

The structure is the paper’s Fig. 3: phoneme embeddings through Tacotron 2’s 3-layer convolutional encoder pre-net and a linear projection (§3.3), plus a scaled positional encoding with a trainable weight α (§3.2, ScaledPositionalEncodingLayer); a Transformer encoder (§3.5); the mel spectrogram through a 2-layer fully connected decoder pre-net and a linear projection plus its own scaled positional encoding (§3.4); a Transformer decoder with masked self-attention and encoder–decoder attention (§3.6, PostNormTransformerDecoderBlock); mel and stop linear projections and a 5-layer convolutional post-net producing a residual (§3.7). Training is teacher-forced and minimizes the mel mean squared error before and after the post-net plus the stop-token binary cross-entropy with a positive weight on the final frame. Inference is autoregressive until the stop token fires.

The model learns its own alignment through attention, so a plain Train(tokens, mel) is its full training signal.

Constructors

Constructor Summary
TransformerTTS(NeuralNetworkArchitecture<T> architecture, TransformerTTSOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
TransformerTTS(NeuralNetworkArchitecture<T> architecture, string modelPath, TransformerTTSOptions options)

Properties

Property Summary
T DecoderPositionScale { get; } The trained scale α of the decoder’s positional encoding.
T EncoderPositionScale { get; } The trained scale α of the encoder’s positional encoding (§3.2, §4.6).
int FftSize { get; }
int HopSize { get; }
int MaxTextLength { get; }
int MelChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> TextToMel(string text) Generates a mel spectrogram from text (the paper pairs it with a WaveNet vocoder).