NaturalSpeech3<T>
Models & Types · AiDotNet.TextToSpeech.CodecBased
NaturalSpeech 3 (Ju et al. 2024): zero-shot speech synthesis with a factorized codec and factorized diffusion. FACodec splits speech into prosody, content and acoustic-detail codes and a timbre vector; masked discrete diffusions generate the durations, prosody, content and detail of new text, each following the corresponding part of a speech prompt; the codec decodes them in the prompt’s timbre.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → NaturalSpeech3<T>
Implements: IEndToEndTts<T>
For Beginners
give it a few seconds of someone speaking, with what they said, and new text; it says the
text in that voice and speaking style. ConvertVoice re-speaks one recording in another’s voice.
How It Works
Inference: the prompt’s codes and timbre from FACodec; the prompt’s and the text’s phonemes through the phoneme encoder; durations by diffusion after the prompt’s (no guidance); then prosody, content and detail by diffusion after the prompt’s, each with classifier-free guidance; FACodec’s decoder on their embeddings with the prompt’s timbre.
Training (CurrentStage): FACodec with its reconstruction, adversarial,
codebook, supervision and gradient-reversal losses; the voice-conversion redecoder; the diffusion model on a frozen codec.
FACodec follows the released code and weights; the diffusion model follows the paper, with MaskGCT’s conventions where
the paper is silent.
Example
var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new NaturalSpeech3<float>(architecture, new NaturalSpeech3Options());
model.LoadFACodec("ns3_facodec_encoder.bin", "ns3_facodec_decoder.bin");
model.Voice = model.CreateVoice(prompt, model.TextToPhonemes("what the prompt says"), promptDurations); // 16 kHz
var audio = model.Synthesize("The quick brown fox.");
Constructors
| Constructor | Summary |
|---|---|
NaturalSpeech3(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
NaturalSpeech3(NeuralNetworkArchitecture<T> architecture, NaturalSpeech3Options options) |
Creates a native, trainable NaturalSpeech 3. |
NaturalSpeech3(NeuralNetworkArchitecture<T> architecture, string modelPath, NaturalSpeech3Options options) |
Creates an ONNX-backed NaturalSpeech 3 for inference. |
Properties
| Property | Summary |
|---|---|
bool TrainsOnSampleObjective { get; } |
|
int MaxTextLength { get; } |
|
int NumFlowSteps { get; } |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> ConvertVoice(Tensor<T> source, Tensor<T> target) |
Zero-shot voice conversion: the source’s prosody and content in the target speaker’s timbre, through the redecoder (IncludeRedecoder) or, without one, FACodec’s decoder; the detail codes are left out, as the release’s examples do. |
Tensor<T> DecodeFactors(int[] codes, Tensor<T> timbre, bool includeDetail) |
FACodec’s waveform of codes [codebooks, frames] in a timbre (the decoder’s inference). |
TtsVoice<T> CreateVoice(Tensor<T> recording, IReadOnlyList<int> phonemes, IReadOnlyList<int> durations) |
A voice from a prompt recording at SampleRate, its phone ids and each phone’s duration in codec frames (the paper’s prompts carry the alignment its duration diffusion follows). |
ValueTuple<int[], Tensor<T>> EncodeFactors(Tensor<T> recording) |
FACodec’s codes of a recording at SampleRate: [codebooks, frames], prosody first, then content, then acoustic detail; and its timbre vector. |
int[] TextToPhonemes(string text) |
The text’s phone ids (ARPAbet; the front end NaturalSpeech 2 uses, see TextToPhonemes). |
void LoadFACodec(string encoderPath, string decoderPath, string redecoderPath) |
Loads Amphion’s released FACodec (amphion/naturalspeech3_facodec): ns3_facodec_encoder.bin and ns3_facodec_decoder.bin (the _v2 files with MelProsody), and optionally ns3_facodec_redecoder.bin (with IncludeRedecoder). |