NaturalSpeech<T>
Models & Types · AiDotNet.TextToSpeech.CodecBased
NaturalSpeech (Tan et al. 2022): end-to-end text to waveform as a VAE whose posterior over the linear spectrogram is simplified and whose prior from the phonemes is enhanced until they match: a pre-trained phoneme encoder, a differentiable durator, a bidirectional prior/posterior flow trained with backward and forward KL losses, a VAE memory bank, and an end-to-end loss on waveforms generated from the prior.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → NaturalSpeech<T>
Implements: IEndToEndTts<T>
For Beginners
NaturalSpeech reads English text aloud in the voice it was trained on, aiming for quality listeners cannot tell from recordings.
How It Works
Inference (Algorithm 1): phonemes → hidden states → durations → the frame-level prior p(z′|y) by learnable upsampling → z′ sampled → z = f(z′) → attention over the memory bank → the HiFi-GAN decoder.
Training follows App. F’s stages (CurrentStage): mixed-phoneme pre-training,
warm-up with monotonic-alignment durations, the joint stage with the memory bank (initialized by K-means of posterior
latents) and both KL directions, and tuning of the durator with the end-to-end loss alone. Details the paper leaves out
come from heatz123/naturalspeech (Heatz123).
Example
var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new NaturalSpeech<float>(architecture, new NaturalSpeechOptions());
// train: Warmup, then model.InitializeMemoryBank(recordings), then Joint, then Tuning
var audio = model.Synthesize("The quick brown fox."); // 22.05 kHz; needs eSpeak NG installed
Constructors
| Constructor | Summary |
|---|---|
NaturalSpeech(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
NaturalSpeech(NeuralNetworkArchitecture<T> architecture, NaturalSpeechOptions options) |
Creates a native, trainable NaturalSpeech. |
NaturalSpeech(NeuralNetworkArchitecture<T> architecture, string modelPath, NaturalSpeechOptions options) |
Creates an ONNX-backed NaturalSpeech for inference. |
Properties
| Property | Summary |
|---|---|
bool TrainsOnSampleObjective { get; } |
|
int MaxTextLength { get; } |
|
int NumFlowSteps { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<ValueTuple<int, int>> LearnSupPhonemes(IEnumerable<int[]> phonemeSequences, int phonemes, int vocabSize) |
Learns sup-phoneme merges (Mixed-Phoneme BERT’s byte-pair encoding of adjacent phonemes inside words) from texts phonemized by phonemize into symbol ids, until vocabSize (the paper: 30,088) units; set the result as SupPhonemeMerges be… |
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
int[] TextToPhonemes(string text) |
The text’s symbol ids: keithito’s english_cleaners2 (ASCII, lower case, abbreviations expanded, whitespace collapsed), eSpeak NG’s phonemes through the phonemizer’s espeak backend with punctuation and stress kept, the symbols looked up, and… |
void InitializeMemoryBank(IEnumerable<Tensor<T>> recordings) |
Initializes the memory bank (App. |
void LoadHeatz123Checkpoint(string path) |
Loads a heatz123/naturalspeech generator checkpoint (G_*.pth’s model); build the model with Heatz123 and the checkpoint’s vocabulary. |