SpeedySpeech<T>

Models & Types · AiDotNet.TextToSpeech.Classic

SpeedySpeech: a fully convolutional non-autoregressive text-to-speech student network trained on durations from a teacher’s attention.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → SpeedySpeech<T>

Implements: IAcousticModel<T>

For Beginners

SpeedySpeech decides how long each phoneme lasts, stretches the phonemes to that many frames, and paints the whole spectrogram at once with stacks of convolutions — no attention, so it is fast.

How It Works

References: “SpeedySpeech: Efficient Neural Speech Synthesis” (Vainer & Dušek, Interspeech 2020), and the authors’ implementation (github.com/janvainer/speedyspeech) for what the paper leaves unstated.

The student (§3.2, Fig. 3) has three convolutional modules built from DilatedResidualConvBlock: a phoneme encoder (embedding, a fully connected layer with ReLU, 13 two-convolution residual blocks with dilations 1, 2, 4 repeated, a long residual connection, and a normalized output projection), a duration predictor reading the encodings with their gradient detached (“we detach gradient flow from the duration predictor to the encoder”), and a decoder (17 blocks with dilations 1, 2, 4, 8 repeated) that turns the expanded encodings, plus a positional encoding restarted at every phoneme, into a log-mel spectrogram. Training uses teacher durations for the expansion and minimizes MAE + (1 − SSIM) on the standardized log-mel spectrogram plus the Huber loss of the log durations.

Training data: durations come from the teacher network (§3.1), so a plain Train(tokens, mel) throws; pass a TtsTrainingSample with Durations.

Constructors

Constructor Summary
SpeedySpeech(NeuralNetworkArchitecture<T> architecture, SpeedySpeechOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
SpeedySpeech(NeuralNetworkArchitecture<T> architecture, string modelPath, SpeedySpeechOptions options)

Properties

Property Summary
int FftSize { get; }
int HopSize { get; }
int MaxTextLength { get; }
int MelChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> TextToMel(string text) Generates a log-mel spectrogram from text (pair it with a vocoder; the paper uses MelGAN).