NaturalSpeech2<T>
Models & Types · AiDotNet.TextToSpeech.CodecBased
NaturalSpeech 2 (Shen et al. 2023): zero-shot speech synthesis as latent diffusion over a neural codec’s continuous quantized vectors, conditioned on a prior of phonemes, durations and pitch, with a speech prompt for in-context learning in the duration/pitch predictors and the diffusion model.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → NaturalSpeech2<T>
Implements: IEndToEndTts<T>
For Beginners
give it a few seconds of someone speaking and some text; it says the text in that voice.
How It Works
Inference: the reference recording’s codec latent → the prompt encoder → query tokens; the phonemes → the phoneme encoder → predicted durations (rounded) → frame-level hidden states + the embedding of the predicted pitch → the condition c; z₁ ~ N(0, τ⁻¹ I) → the reverse ODE solved with the WaveNet denoiser predicting ẑ₀ → the codec’s decoder.
Training (Train): a random span of the utterance becomes the
prompt and the rest the target; L = L_diff + L_dur + L_pitch with the ground-truth durations and pitch in the prior.
The codec is trained first, on its own (Codec). Details the paper leaves out come from Amphion’s
implementation, whose departures are options (Amphion).
Example
var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new NaturalSpeech2<float>(architecture, new NaturalSpeech2Options());
// train model.Codec on recordings, then model.Train(new TtsTrainingSample<float> { Tokens = ids, Durations = frames, Audio = wave })
model.Voice = model.CreateVoice(reference); // 16 kHz
var audio = model.Synthesize("The quick brown fox.");
Constructors
| Constructor | Summary |
|---|---|
NaturalSpeech2(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
NaturalSpeech2(NeuralNetworkArchitecture<T> architecture, NaturalSpeech2Options options) |
Creates a native, trainable NaturalSpeech 2. |
NaturalSpeech2(NeuralNetworkArchitecture<T> architecture, string modelPath, NaturalSpeech2Options options) |
Creates an ONNX-backed NaturalSpeech 2 for inference. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> Symbols { get; } |
The phone set (Amphion’s): CMUdict’s 84 ARPAbet symbols, then sp, spn, sil, |
NeuralAudioCodecBase<T> Codec { get; } |
The codec (Codec): a model of its own. |
bool TrainsOnSampleObjective { get; } |
|
int CodecTokenCodebooks { get; } |
|
int CodecTokenVocabulary { get; } |
|
int MaxTextLength { get; } |
|
int NumFlowSteps { get; } |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
TtsVoice<T> CreateVoice(Tensor<T> recording) |
A voice from a recording of the speaker at SampleRate (the speech prompt). |
int[] TextToPhonemes(string text) |
The text’s phone ids (Amphion’s preprocess_english): trailing punctuation stripped, the text split at punctuation and spaces, each piece looked up in the lexicon (LexiconPath) or pronounced by g2p_en, a punctuation mark becoming sp. |
void LoadAmphionCheckpoint(string path) |
Loads Amphion’s NaturalSpeech 2 checkpoint (amphion/naturalspeech2_libritts’s pytorch_model.bin). |