NaturalSpeech3<T>

Models & Types · AiDotNet.TextToSpeech.CodecBased

NaturalSpeech 3 (Ju et al. 2024): zero-shot speech synthesis with a factorized codec and factorized diffusion. FACodec splits speech into prosody, content and acoustic-detail codes and a timbre vector; masked discrete diffusions generate the durations, prosody, content and detail of new text, each following the corresponding part of a speech prompt; the codec decodes them in the prompt’s timbre.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → NaturalSpeech3<T>

Implements: IEndToEndTts<T>

For Beginners

give it a few seconds of someone speaking, with what they said, and new text; it says the text in that voice and speaking style. ConvertVoice re-speaks one recording in another’s voice.

How It Works

Inference: the prompt’s codes and timbre from FACodec; the prompt’s and the text’s phonemes through the phoneme encoder; durations by diffusion after the prompt’s (no guidance); then prosody, content and detail by diffusion after the prompt’s, each with classifier-free guidance; FACodec’s decoder on their embeddings with the prompt’s timbre.

Training (CurrentStage): FACodec with its reconstruction, adversarial, codebook, supervision and gradient-reversal losses; the voice-conversion redecoder; the diffusion model on a frozen codec. FACodec follows the released code and weights; the diffusion model follows the paper, with MaskGCT’s conventions where the paper is silent.

Example

var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new NaturalSpeech3<float>(architecture, new NaturalSpeech3Options());
model.LoadFACodec("ns3_facodec_encoder.bin", "ns3_facodec_decoder.bin");
model.Voice = model.CreateVoice(prompt, model.TextToPhonemes("what the prompt says"), promptDurations);   // 16 kHz
var audio = model.Synthesize("The quick brown fox.");

Constructors

Constructor Summary
NaturalSpeech3(NeuralNetworkArchitecture<T> architecture) Calls the fully specified overload with the omitted parameters’ defaults.
NaturalSpeech3(NeuralNetworkArchitecture<T> architecture, NaturalSpeech3Options options) Creates a native, trainable NaturalSpeech 3.
NaturalSpeech3(NeuralNetworkArchitecture<T> architecture, string modelPath, NaturalSpeech3Options options) Creates an ONNX-backed NaturalSpeech 3 for inference.

Properties

Property Summary
bool TrainsOnSampleObjective { get; }
int MaxTextLength { get; }
int NumFlowSteps { get; }

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> ConvertVoice(Tensor<T> source, Tensor<T> target) Zero-shot voice conversion: the source’s prosody and content in the target speaker’s timbre, through the redecoder (IncludeRedecoder) or, without one, FACodec’s decoder; the detail codes are left out, as the release’s examples do.
Tensor<T> DecodeFactors(int[] codes, Tensor<T> timbre, bool includeDetail) FACodec’s waveform of codes [codebooks, frames] in a timbre (the decoder’s inference).
TtsVoice<T> CreateVoice(Tensor<T> recording, IReadOnlyList<int> phonemes, IReadOnlyList<int> durations) A voice from a prompt recording at SampleRate, its phone ids and each phone’s duration in codec frames (the paper’s prompts carry the alignment its duration diffusion follows).
ValueTuple<int[], Tensor<T>> EncodeFactors(Tensor<T> recording) FACodec’s codes of a recording at SampleRate: [codebooks, frames], prosody first, then content, then acoustic detail; and its timbre vector.
int[] TextToPhonemes(string text) The text’s phone ids (ARPAbet; the front end NaturalSpeech 2 uses, see TextToPhonemes).
void LoadFACodec(string encoderPath, string decoderPath, string redecoderPath) Loads Amphion’s released FACodec (amphion/naturalspeech3_facodec): ns3_facodec_encoder.bin and ns3_facodec_decoder.bin (the _v2 files with MelProsody), and optionally ns3_facodec_redecoder.bin (with IncludeRedecoder).