NaturalSpeech2Options

Options & Configuration · AiDotNet.TextToSpeech.CodecBased

Options for NaturalSpeech2.

Inheritance: ModelOptions → TtsModelOptions → NaturalSpeech2Options

For Beginners

NaturalSpeech 2 speaks in the voice of a few seconds of reference audio. A neural codec turns speech into continuous vectors; a diffusion model, told the phonemes, how long each lasts and the pitch, generates those vectors for new text; the codec turns them back into audio. Train the codec first (see Codec), then the diffusion model.

How It Works

The defaults are the paper’s (Shen et al. 2023, §4.1, §5.1, Table 11, App. B). Where it is silent the values come from Amphion’s implementation (open-mmlab/Amphion, models/tts/naturalspeech2), and where Amphion departs from the paper its choice is an option; Amphion selects all of them, which its released checkpoint needs.

Constructors

Constructor Summary
NaturalSpeech2Options() Creates the paper’s configuration.

Properties

Property Summary
NaturalSpeech2CeResidual CeResidual { get; set; } The residual the RVQ cross-entropy reads (the paper’s ground truth; Amphion’s prediction).
NaturalSpeech2Diffusion Diffusion { get; set; } The diffusion formulation (the paper’s score SDE; Amphion’s flow matching is an option).
NaturalSpeech2Positions Positions { get; set; } The Transformers’ positional encoding (Sinusoidal; Amphion’s batch-indexed table).
NaturalSpeech2PromptPosition PromptPosition { get; set; } Where the training prompt is cut (anywhere; Amphion’s configured start).
NaturalSpeech2ScoreLoss ScoreLoss { get; set; } Eq.
NaturalSpeech2Solver Solver { get; set; } The ODE solver (Euler).
NeuralAudioCodecOptions Codec { get; set; } The neural audio codec whose quantized latents the diffusion model generates (PaperCodec, or AmphionCodec).
bool CosineSchedule { get; set; } Whether the learning rate follows a cosine decay to zero at TrainingSteps after the warm-up (false: the inverse square root; Amphion’s cosine).
bool DataLossL1 { get; set; } Whether the data term is an L1 distance (false: Eq.
bool NoiseLossL1 { get; set; } Whether the score term is an L1 distance (false; Amphion’s L1).
bool StyleAdaptivePhonemeEncoder { get; set; } Whether the phoneme encoder’s LayerNorms are style-adaptive on the prompt (false; Amphion’s true).
double BetaMax { get; set; } β at t = 1 (20; Amphion’s).
double BetaMin { get; set; } β at t = 0 of the linear noise schedule (0.05; Amphion’s, the paper’s schedule is unstated).
double CeLossWeight { get; set; } λ_ce-rvq (0.1, §3.2; Amphion’s 0.5).
double DenoiserAttentionDropout { get; set; } Dropout on the denoiser’s attention weights (0.1, Amphion’s).
double GradientClipNorm { get; set; } The gradient-norm clip (0.5; Amphion’s, the paper’s is unstated).
double NoiseLossWeight { get; set; } The score term’s weight (1).
double PitchMax { get; set; } The highest pitch bucket boundary in Hz (1100; Amphion’s).
double PitchMin { get; set; } The lowest pitch bucket boundary in Hz (50; Amphion’s).
double PredictorDropout { get; set; } The predictors’ dropout (0.5).
double PromptMaxFraction { get; set; } Its largest share (0.5; Amphion’s).
double PromptMinFraction { get; set; } The training prompt’s smallest share of the phonemes (0.1; Amphion’s).
double Temperature { get; set; } The sampling temperature τ: z₁ ~ N(0, τ⁻¹ I) (1.22; Amphion divides by 1.2, τ = 1.44).
double TimeEpsilon { get; set; } Training steps t are drawn from U(0, 1) clamped to [ε, 1 − ε] (1e-5; Amphion’s).
int DenoiserAttentionEvery { get; set; } The denoiser attends to the query tokens and applies FiLM every this-many layers (3).
int DenoiserAttentionOffset { get; set; } The first such layer, 0-based (2: layers 3, 6 …
int DenoiserDilation { get; set; } The denoiser’s dilation (2, App.
int DenoiserLayers { get; set; } The denoiser’s WaveNet layers (40).
int DurationKernel { get; set; } The duration predictor’s kernel (3).
int DurationLayers { get; set; } The duration predictor’s convolution layers (30).
int EncoderKernel { get; set; } Their feed-forward convolution’s kernel (9).
int FilterChannels { get; set; } The phoneme encoder’s and the prompt encoder’s feed-forward filter (2048).
int NumPromptLayers { get; set; } The speech prompt encoder’s Transformer layers (6).
int PitchBins { get; set; } The pitch embedding’s buckets (512; Amphion’s), log-spaced between the boundaries.
int PitchKernel { get; set; } The pitch predictor’s kernel (5).
int PitchLayers { get; set; } The pitch predictor’s convolution layers (30).
int PredictorAttentionEvery { get; set; } The predictors attend to the prompt before every this-many convolutions (3: ten attention layers of 30).
int QueryTokens { get; set; } The prompt’s query tokens m (32).
int SamplingSteps { get; set; } The reverse ODE’s steps (150; Amphion’s inference script 200).
int TrainingSteps { get; set; } The cosine schedule’s length (800000; Amphion’s).
int WarmupSteps { get; set; } The learning-rate warm-up steps (32000; Amphion’s 5000).
int? DenoiserDilationCycle { get; set; } A dilation cycle c giving layer i the dilation 2^(i mod c) (null; Amphion’s 2: 1, 2, 1, 2 …).
string LexiconPath { get; set; } A pronunciation lexicon (word, then ARPAbet phones, per line; Amphion’s librispeech-lexicon.txt) consulted before g2p_en, as Amphion’s front end does; null uses g2p_en alone.

Methods

Method Summary
EnCodecOptions AmphionCodec() Amphion’s codec: EnCodec’s released 24 kHz model at 12 kbps (16 codebooks of dimension 128).
NaturalSpeech2Options Amphion() Amphion’s configuration (config/ns2.json, egs/tts/NaturalSpeech2/exp_config.json) and every departure of its code from the paper, as its LibriTTS checkpoint (amphion/naturalspeech2_libritts) was trained.
SoundStreamOptions PaperCodec() The paper’s codec (§3.1, Table 11): SoundStream’s model at 16 kHz with a total stride of 200 (12.5 ms frames; strides 2, 4, 5, 5), 16 residual quantizers of 1024 codes of dimension 256, without the denoising FiLM.