ForwardTacotron<T>

Models & Types · AiDotNet.TextToSpeech.Classic

ForwardTacotron: a non-autoregressive Tacotron that predicts phoneme durations, pitch and energy and generates the whole mel spectrogram in one pass.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → ForwardTacotron<T>

Implements: IAcousticModel<T>

For Beginners

ForwardTacotron decides how long, how high and how loud each sound is, then paints the whole spectrogram at once, so it never skips or repeats words.

How It Works

Reference: ForwardTacotron (Axel Springer AI, as-ideas/ForwardTacotron), a model published as a reference implementation rather than a paper; it is inspired by FastSpeech and built from Tacotron components. The autoregressive Non-Attentive Tacotron (Shen et al. 2020) is a different model.

Three series predictors (phoneme embedding, three kernel-5 convolutions with ReLU and batch normalization, a bidirectional GRU, a linear projection) predict each phoneme’s duration in frames and its normalized pitch and energy. The mel path embeds the phonemes, runs a CBHG pre-net (ForwardTacotron), adds the pitch and energy series through kernel-3 convolutions, expands by the durations, and decodes with a bidirectional LSTM, a linear projection and a CBHG post-net with a bias-free projection. Training uses the ground-truth durations (from a Tacotron’s attention), phoneme-level pitch (mean voiced WORLD F0 in 30–600 Hz) and energy (mean L2 norm of exp(mel)), with L1 losses on mel, post-net mel and 0.1 × each series.

Durations come from a forced alignment outside the model, so a plain Train(tokens, mel) throws; pass a TtsTrainingSample with Durations.

Constructors

Constructor Summary
ForwardTacotron(NeuralNetworkArchitecture<T> architecture, ForwardTacotronOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
ForwardTacotron(NeuralNetworkArchitecture<T> architecture, string modelPath, ForwardTacotronOptions options)

Properties

Property Summary
int FftSize { get; }
int HopSize { get; }
int MaxTextLength { get; }
int MelChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> TextToMel(string text) Generates a log-mel spectrogram from text (pair it with a vocoder).