NaturalSpeech<T>

Models & Types · AiDotNet.TextToSpeech.CodecBased

NaturalSpeech (Tan et al. 2022): end-to-end text to waveform as a VAE whose posterior over the linear spectrogram is simplified and whose prior from the phonemes is enhanced until they match: a pre-trained phoneme encoder, a differentiable durator, a bidirectional prior/posterior flow trained with backward and forward KL losses, a VAE memory bank, and an end-to-end loss on waveforms generated from the prior.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → NaturalSpeech<T>

Implements: IEndToEndTts<T>

For Beginners

NaturalSpeech reads English text aloud in the voice it was trained on, aiming for quality listeners cannot tell from recordings.

How It Works

Inference (Algorithm 1): phonemes → hidden states → durations → the frame-level prior p(z′|y) by learnable upsampling → z′ sampled → z = f(z′) → attention over the memory bank → the HiFi-GAN decoder.

Training follows App. F’s stages (CurrentStage): mixed-phoneme pre-training, warm-up with monotonic-alignment durations, the joint stage with the memory bank (initialized by K-means of posterior latents) and both KL directions, and tuning of the durator with the end-to-end loss alone. Details the paper leaves out come from heatz123/naturalspeech (Heatz123).

Example

var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new NaturalSpeech<float>(architecture, new NaturalSpeechOptions());
// train: Warmup, then model.InitializeMemoryBank(recordings), then Joint, then Tuning
var audio = model.Synthesize("The quick brown fox.");   // 22.05 kHz; needs eSpeak NG installed

Constructors

Constructor Summary
NaturalSpeech(NeuralNetworkArchitecture<T> architecture) Calls the fully specified overload with the omitted parameters’ defaults.
NaturalSpeech(NeuralNetworkArchitecture<T> architecture, NaturalSpeechOptions options) Creates a native, trainable NaturalSpeech.
NaturalSpeech(NeuralNetworkArchitecture<T> architecture, string modelPath, NaturalSpeechOptions options) Creates an ONNX-backed NaturalSpeech for inference.

Properties

Property Summary
bool TrainsOnSampleObjective { get; }
int MaxTextLength { get; }
int NumFlowSteps { get; }

Methods

Method Summary
IReadOnlyList<ValueTuple<int, int>> LearnSupPhonemes(IEnumerable<int[]> phonemeSequences, int phonemes, int vocabSize) Learns sup-phoneme merges (Mixed-Phoneme BERT’s byte-pair encoding of adjacent phonemes inside words) from texts phonemized by phonemize into symbol ids, until vocabSize (the paper: 30,088) units; set the result as SupPhonemeMerges be…
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
int[] TextToPhonemes(string text) The text’s symbol ids: keithito’s english_cleaners2 (ASCII, lower case, abbreviations expanded, whitespace collapsed), eSpeak NG’s phonemes through the phonemizer’s espeak backend with punctuation and stress kept, the symbols looked up, and…
void InitializeMemoryBank(IEnumerable<Tensor<T>> recordings) Initializes the memory bank (App.
void LoadHeatz123Checkpoint(string path) Loads a heatz123/naturalspeech generator checkpoint (G_*.pth’s model); build the model with Heatz123 and the checkpoint’s vocabulary.