NaturalSpeech3Options

Options & Configuration · AiDotNet.TextToSpeech.CodecBased

Options for NaturalSpeech3.

Inheritance: ModelOptions → TtsModelOptions → NaturalSpeech3Options

For Beginners

NaturalSpeech 3 speaks text in the voice and style of a short prompt. Its codec splits speech into prosody, content, acoustic detail and timbre; diffusion models generate the first three for the text, each following the prompt’s own, and the codec puts them back together in the prompt’s timbre. Train the codec first, then the diffusion model.

How It Works

FACodec follows the released code and checkpoints (amphion/naturalspeech3_facodec) with the paper’s training (App. B.1); the factorized diffusion model follows the paper (§3.3, App. A), with MaskGCT’s conventions where the paper is silent. Where the release and the paper differ, the other is an option.

Constructors

Constructor Summary
NaturalSpeech3Options() Creates the configuration.

Properties

Property Summary
NaturalSpeech3Stage CurrentStage { get; set; } What the training step trains (Diffusion).
bool ContentF0Reversal { get; set; } The F0-reversal head on z_c, which the paper describes and the release leaves off (false).
bool DetachProsodyContent { get; set; } Whether the decoder reads stop(z_p) + stop(z_c) during training, as the release’s forward does (true); false trains the quantizers through the reconstruction too.
bool DetailF0Reversal { get; set; } The F0-reversal head on z_d (true).
bool DetailPhoneReversal { get; set; } The phone-reversal head on z_d (true).
bool IncludeRedecoder { get; set; } Whether the voice-conversion redecoder is built (false).
bool MelProsody { get; set; } FACodec v2: prosody quantized from the first 20 mel bins rather than the encoder latent (false).
bool PhonemeProsody { get; set; } The phoneme-level prosody diffusion the paper’s inference lists but never defines (false).
bool ProsodyPhoneReversal { get; set; } The phone-reversal head on z_p, which the paper describes and the release leaves off (false).
bool TimbreReversal { get; set; } The speaker-reversal head on z_p + z_c + z_d (true).
double AdversarialWeight { get; set; } λ_adv (2).
double CodebookWeight { get; set; } λ_codebook (1).
double CodecBeta1 { get; set; } The codec’s Adam β1 (0.5).
double CodecBeta2 { get; set; } The codec’s Adam β2 (0.9).
double CodecLearningRate { get; set; } The codec’s Adam learning rate (2e-4).
double CommitmentWeight { get; set; } λ_commit (0.25).
double DetailDropout { get; set; } The detail dropout’s probability of masking z_d during training (0.75, the release’s; the paper’s p).
double DiffusionBeta2 { get; set; } The diffusion’s AdamW β2 (0.98; β1 0.9).
double F0ReversalWeight { get; set; } λ_gr-f0 (5).
double F0Weight { get; set; } λ_f0 (5).
double FeatureWeight { get; set; } λ_feat (2).
double GuidanceScale { get; set; } The classifier-free guidance scale α for prosody, content and detail (1.0; durations use none).
double PhoneReversalWeight { get; set; } λ_gr-ph (5).
double PhoneWeight { get; set; } λ_ph (5).
double PromptDropProbability { get; set; } p_cfg, the prompt’s drop probability in training (0.15).
double ReconstructionWeight { get; set; } λ_rec (10).
double SpeakerReversalWeight { get; set; } λ_gr-spk (1).
double Temperature { get; set; } The sampling temperature at the first iteration, annealed to 0 (1.5).
int CodebookDim { get; set; } The factorized codes’ dimension, the information bottleneck (8).
int CodebookSize { get; set; } The codes per codebook (1024).
int CodecDecoderChannels { get; set; } The decoder’s first width (1024).
int CodecDim { get; set; } The latent width (256).
int CodecEncoderDim { get; set; } The encoder’s first width (32).
int CodecPhones { get; set; } The phone classes the content supervision predicts (5003, the release’s).
int CodecSegment { get; set; } The codec’s training clip in samples (16000).
int CodecSpeakers { get; set; } The speakers the speaker-reversal head classifies (245200, the release’s).
int ContentQuantizers { get; set; } The content quantizers N_qc (2).
int DetailQuantizers { get; set; } The acoustic-detail quantizers N_qd (3).
int DiffusionDim { get; set; } The diffusion Transformers’ width (1024).
int DiffusionFilter { get; set; } Their feed-forward filter (2048).
int DiffusionHeads { get; set; } Their attention heads (8).
int DiffusionKernel { get; set; } Their feed-forward kernel (3).
int DiscriminatorWidthDivisor { get; set; } A divisor on every discriminator width (1).
int EncoderKernel { get; set; } The phoneme encoder’s feed-forward kernel (9).
int FilterChannels { get; set; } The phoneme encoder’s feed-forward filter (2048).
int FrameDiffusionLayers { get; set; } The shared prosody/content/detail Transformer’s layers (12).
int HopSize { get; set; } The codec hop, which must equal both stride products.
int MaxDuration { get; set; } The longest duration token in frames (63: durations are 64 classes, longer ones clamped).
int MelChannels { get; set; } Inherited mel-output metadata; NaturalSpeech 3 decodes waveforms through FACodec.
int PhonemeDiffusionLayers { get; set; } The duration (and phoneme-level prosody) Transformer’s layers (6).
int ProsodyQuantizers { get; set; } The prosody quantizers N_qp (1).
int SamplingSteps { get; set; } The iterations of each diffusion (4).
int TopK { get; set; } The top-k filter (20).
int WarmupSteps { get; set; } The diffusion’s warm-up steps before the inverse square root (5000).
int[] CodecDecoderRates { get; set; } The decoder’s strides (5, 5, 4, 2).
int[] CodecEncoderRates { get; set; } The encoder’s strides (2, 4, 5, 5: a hop of 200 at 16 kHz).
int[] DiscriminatorPeriods { get; set; } The multi-period discriminator’s periods (2, 3, 5, 7, 11).
int[] DiscriminatorWindows { get; set; } The multi-band multi-scale STFT discriminator’s windows (2048, 1024, 512).
int[] MelBins { get; set; } The multi-scale mel loss’s mel bins (DAC’s 5 …
int[] MelWindows { get; set; } The multi-scale mel loss’s windows (DAC’s 32 …
string LexiconPath { get; set; } A pronunciation lexicon consulted before g2p_en (see LexiconPath).