DeepVoice3<T>

Models & Types · AiDotNet.TextToSpeech.Classic

Deep Voice 3: fully convolutional attention-based text-to-speech.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → DeepVoice3<T>

Implements: IAcousticModel<T>

For Beginners

Deep Voice 3 replaces recurrent networks with stacks of convolutions, so it trains fast, and uses attention to decide which letters it is speaking at each step.

How It Works

References: “Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning” (Ping et al., ICLR 2018), single-speaker configuration with the Griffin–Lim converter; unstated details follow the reference implementation r9y9/deepvoice3_pytorch.

The encoder (§3.4) projects character embeddings to the convolution width, applies non-causal gated convolution blocks (GatedConvolutionBlockLayer) and projects back to attention keys; values are √0.5 (keys + embeddings). The decoder (§3.5) reads the previous group of r frames through a fully connected pre-net, then alternates causal convolution blocks with attention blocks (§3.6: positional encodings with rates ω_query and ω_key added to queries and keys, dot-product softmax attention, context normalization, √0.5 residuals), and predicts the next r frames and a done flag. The converter (§3.7) maps the decoder’s hidden states to the linear spectrogram. Training minimizes L1 on mel, binary cross-entropy on done and L1 on the linear spectrogram, with gradients clipped to norm 100 and value 5 (Table 4). Inference is autoregressive with attention restricted to a window around the last attended position and stops on the done flag.

Constructors

Constructor Summary
DeepVoice3(NeuralNetworkArchitecture<T> architecture, DeepVoice3Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
DeepVoice3(NeuralNetworkArchitecture<T> architecture, string modelPath, DeepVoice3Options options)

Properties

Property Summary
int FftSize { get; }
int HopSize { get; }
int MaxTextLength { get; }
int MelChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> PredictLinearSpectrogram(string text) Predicts the log-magnitude linear spectrogram for text, the converter’s output.
Tensor<T> TextToMel(string text) Generates a log-mel spectrogram from text.