MegaTTS2Options

Options & Configuration · AiDotNet.TextToSpeech.Latest

Options for MegaTTS2. The sizes are the paper’s (Jiang et al. 2024, Appendix A.4 and Table 5); what it leaves unstated comes from the authors’ code (NATSpeech, SyntaSpeech, MegaTTS 3’s release) and from CVQ-VAE.

Inheritance: ModelOptions → TtsModelOptions → MegaTTS2Options

For Beginners

Mega-TTS 2 speaks in a voice it has never heard from a few seconds, or a few minutes, of it: the more prompt sentences you give it, the closer it gets to the speaker. These options size its parts; train it on recordings with phoneme durations.

Constructors

Constructor Summary
MegaTTS2Options() Creates an instance with the default settings.

Properties

Property Summary
MegaTts2CodeAnchor CodeAnchor { get; set; } Where the dynamic initialization draws its anchors (CVQ-VAE’s closest features).
MegaTts2CodeDistance CodeDistance { get; set; } How a vector finds its code (CVQ-VAE’s cosine similarity).
MegaTts2Stage Stage { get; set; } The stage MegaTTS2 trains.
MegaTtsLanguage Language { get; set; } The text front end (English; the paper trains on LibriLight).
double AdversarialWeight { get; set; } The adversarial loss weight (SyntaSpeech’s 0.05).
double Beta1 { get; set; } Adam’s β1 (0.9), β2 (0.999) and ε (1e-9), on the Transformer schedule (§4.1).
double Beta2 { get; set; } See Beta1.
double CodeUsageDecay { get; set; } The decay of the codes’ running usage (CVQ-VAE’s 0.99).
double CommitmentWeight { get; set; } The VQ loss’s commitment weight β (0.25, van den Oord et al.
double DiscriminatorDecay { get; set; } See DiscriminatorLearningRate.
double DiscriminatorEpsilon { get; set; } See DiscriminatorLearningRate.
double DiscriminatorLearningRate { get; set; } The discriminator’s learning rate (1e-4), ε (1e-6) and step decay (halved every 40000 steps), as SyntaSpeech trains it.
double Epsilon { get; set; } See Beta1.
double MaxGradientNorm { get; set; } The gradient norm every step clips to (NATSpeech’s clip_grad_norm of 1); 0 turns clipping off.
double MelMaxFrequency { get; set; } The mel filterbank’s highest frequency (NATSpeech’s 7600 Hz).
double MelMinFrequency { get; set; } The mel filterbank’s lowest frequency (NATSpeech’s 80 Hz).
double ProsodyInterpolation { get; set; } Prosody interpolation’s weight γ on the auxiliary prosody prompt (0.8 in the paper’s transfer experiments, Eq.
double ScheduleFactor { get; set; } The Transformer schedule’s scale (1: d^−½ · min(t^−½, t · warmup^−1.5)).
int CodeDimension { get; set; } See Codes.
int Codes { get; set; } The codebook’s entries (1024) and their width (256).
int Compression { get; set; } The VQ encoder’s compression rate r: the max pooling of the mel spectrogram (8).
int ContentFilter { get; set; } See ContentLayers.
int ContentHeads { get; set; } See ContentLayers.
int ContentLayers { get; set; } The content encoder’s layers (8), heads (NATSpeech’s 2) and feed-forward filter (1024); kernel 5.
int DecoderBlocks { get; set; } The mel decoder’s blocks (4 blocks of 2 layers, kernel 5).
int DiscriminatorDecaySteps { get; set; } See DiscriminatorLearningRate.
int DiscriminatorHidden { get; set; } See DiscriminatorWindows.
int Downsampling { get; set; } See TimbreHidden.
int DurationCodes { get; set; } The largest duration the duration model reads as a token; longer ones read as it (MegaTTS 3’s 128).
int DurationHeads { get; set; } See DurationLayers.
int DurationHidden { get; set; } See DurationLayers.
int DurationLayers { get; set; } The autoregressive duration model’s layers (8), width (512) and heads (8, MegaTTS 3’s).
int LmHeads { get; set; } See LmLayers.
int LmHidden { get; set; } See LmLayers.
int LmLayers { get; set; } The P-LLM’s layers (12), width (1024) and heads (16).
int NumDecoderLayers { get; set; } Gets or sets the mel decoder’s convolutional block count.
int NumEncoderLayers { get; set; } Gets or sets the FastSpeech content encoder depth.
int NumHeads { get; set; } Gets or sets the FastSpeech content encoder attention head count.
int ProsodyBlocks { get; set; } See ProsodyHidden.
int ProsodyHidden { get; set; } The VQ prosody encoder’s width (384) and blocks per stack (3 blocks of 2 layers, kernel 5).
int ProsodyTokenBudget { get; set; } The most prosody tokens one language-model step reads (the paper’s 4000 per GPU).
int SamplingSeed { get; set; } The seed of the prosody sampling.
int TimbreBlocks { get; set; } See TimbreHidden.
int TimbreHeads { get; set; } The timbre-to-content attention’s heads (NATSpeech’s 2).
int TimbreHidden { get; set; } The MRTE’s acoustic width (the key encoder’s 256), blocks per stack (5 blocks of 2 layers, kernel 3) and downsampling rate d (16).
int TimbreReferenceFrames { get; set; } The frames of the other recordings the MRTE reads in training, drawn at random (2000).
int TopK { get; set; } Top-k sampling of the prosody codes (10, Appendix A.6).
int TrainingSeed { get; set; } The seed of training’s draws (the discriminator windows, the MRTE’s frames, CVQ-VAE’s anchors).
int VocoderInitialChannels { get; set; } HiFi-GAN V1’s first width (512), upsampling rates and kernels, and residual-block kernels and dilations (Kong et al.
int WarmupSteps { get; set; } The Transformer schedule’s warm-up steps (Vaswani et al.’s 4000).
int WindowSize { get; set; } The analysis window, in samples (NATSpeech’s 1024).
int[] DiscriminatorWindows { get; set; } The multi-length discriminator’s windows (32, 64, 128 frames) and width (192).
int[] VocoderKernels { get; set; } See VocoderInitialChannels.
int[] VocoderRates { get; set; } See VocoderInitialChannels.
int[] VocoderResBlockKernels { get; set; } See VocoderInitialChannels.
int[][] VocoderResBlockDilations { get; set; } See VocoderInitialChannels.
string[] PhoneSet { get; set; } The phone set (NATSpeech’s ph_set.json): its sorted phones follow the three reserved tokens.