DeepVoice3<T>
Models & Types · AiDotNet.TextToSpeech.Classic
Deep Voice 3: fully convolutional attention-based text-to-speech.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → DeepVoice3<T>
Implements: IAcousticModel<T>
For Beginners
Deep Voice 3 replaces recurrent networks with stacks of convolutions, so it trains fast, and uses attention to decide which letters it is speaking at each step.
How It Works
References: “Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning” (Ping et al., ICLR 2018), single-speaker configuration with the Griffin–Lim converter; unstated details follow the reference implementation r9y9/deepvoice3_pytorch.
The encoder (§3.4) projects character embeddings to the convolution width, applies non-causal gated convolution
blocks (GatedConvolutionBlockLayer) and projects back to attention keys; values are
√0.5 (keys + embeddings). The decoder (§3.5) reads the previous group of r frames through a fully connected pre-net,
then alternates causal convolution blocks with attention blocks (§3.6: positional encodings with rates ω_query and
ω_key added to queries and keys, dot-product softmax attention, context normalization, √0.5 residuals), and predicts
the next r frames and a done flag. The converter (§3.7) maps the decoder’s hidden states to the linear spectrogram.
Training minimizes L1 on mel, binary cross-entropy on done and L1 on the linear spectrogram, with gradients clipped
to norm 100 and value 5 (Table 4). Inference is autoregressive with attention restricted to a window around the
last attended position and stops on the done flag.
Constructors
| Constructor | Summary |
|---|---|
DeepVoice3(NeuralNetworkArchitecture<T> architecture, DeepVoice3Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
DeepVoice3(NeuralNetworkArchitecture<T> architecture, string modelPath, DeepVoice3Options options) |
Properties
| Property | Summary |
|---|---|
int FftSize { get; } |
|
int HopSize { get; } |
|
int MaxTextLength { get; } |
|
int MelChannels { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> PredictLinearSpectrogram(string text) |
Predicts the log-magnitude linear spectrogram for text, the converter’s output. |
Tensor<T> TextToMel(string text) |
Generates a log-mel spectrogram from text. |