SpeechT5Options

Options & Configuration · AiDotNet.TextToSpeech.MultiModal

Options for SpeechT5.

Inheritance: ModelOptions → TtsModelOptions → SpeechT5Options

For Beginners

SpeechT5 is one encoder-decoder pre-trained on both speech and text and fine-tuned to read text aloud in the voice of a speaker embedding (an x-vector computed from a recording). These settings size the networks and control training and generation.

How It Works

The defaults are the paper’s base model (Ao et al. 2022: 12 encoder and 6 decoder layers of width 768, 12 heads, feed-forward 3072, relative position embeddings; 100 × 2 codebook entries mixed into 10% of the states) with the official recipe filling what the paper leaves out (t5_transformer_base, the pre-training and TTS commands), and the paper’s training rates: pre-training warms up over 8% of 500k updates to 2e-4 then decays linearly; text-to-speech fine-tuning warms up over 10k of 120k steps to 4e-4 then decays by the inverse square root. The released weights (microsoft/speecht5_tts) use these sizes.

Constructors

Constructor Summary
SpeechT5Options() Creates the paper’s base configuration.

Properties

Property Summary
SpeechT5Stage CurrentStage { get; set; } The objective SpeechT5 trains (text-to-speech fine-tuning).
SpeechT5StopCriterion StopCriterion { get; set; } When generation stops (the official code’s: any frame).
ValueTuple<int, int, int>[] FeatureLayers { get; set; } The speech encoder pre-net’s convolutions as (channels, kernel, stride): wav2vec 2.0’s seven.
bool BertInitialization { get; set; } Whether a fresh model draws its weights as the official pre-training’s --bert-init does (N(0, 0.02) linear layers and embeddings, zero biases; true).
double ActivationDropout { get; set; } Dropout after the feed-forward activation (0.15, the TTS recipe’s).
double AttentionDropout { get; set; } Attention-weight dropout (0.15, the TTS recipe’s).
double CodebookProbability { get; set; } Fraction of encoder states replaced by their quantized vectors (0.1).
double DecoderLayerDrop { get; set; } LayerDrop of the decoder layers (0 in the TTS recipe; 0.05 in pre-training).
double DiversityWeight { get; set; } Weight of the codebook diversity loss (0.1).
double EncoderLayerDrop { get; set; } LayerDrop of the encoder layers (0 in the TTS recipe; 0.05 in pre-training).
double FeaturePenaltyWeight { get; set; } Weight of the feature penalty (10).
double GuidedAttentionSigma { get; set; } The guided attention loss’s σ (0.4).
double LabelRate { get; set; } HuBERT labels per second (50).
double MaskProbability { get; set; } The span masking’s start probability (0.8 → ≈8% of frames start a span, §2.3).
double MaxGradientNorm { get; set; } Text-to-speech’s gradient-norm clip (25, the recipe’s).
double MaxLengthRatio { get; set; } The longest generation, in decoder steps per text token times r (20).
double OptimizerBeta1 { get; set; } Adam β1 (0.9).
double OptimizerBeta2 { get; set; } Adam β2 (0.98).
double OptimizerEpsilon { get; set; } Text-to-speech’s Adam ε (1e-8, the recipe’s default).
double PoissonLambda { get; set; } The Poisson λ of the infilling’s span lengths (3.5).
double PositionalDropout { get; set; } Dropout after the scaled positional encodings (0.1).
double PostnetDropout { get; set; } The post-net’s dropout (0.5).
double PrenetDropout { get; set; } The speech decoder pre-net’s always-on dropout (0.5, Tacotron 2’s).
double PretrainEpsilon { get; set; } Pre-training’s Adam ε (1e-6, the recipe’s).
double PretrainLearningRate { get; set; } Pre-training’s peak learning rate (2e-4, §3.1).
double PretrainMaxGradientNorm { get; set; } Pre-training’s gradient-norm clip (5, the recipe’s).
double PretrainWarmupFraction { get; set; } Fraction of pre-training updates spent warming up (0.08).
double PretrainWeightDecay { get; set; } Pre-training’s decoupled weight decay (0.01, the recipe’s).
double StopPositiveWeight { get; set; } Weight of the stop token’s positive class in its BCE (5).
double StopThreshold { get; set; } Stop probability that ends generation (0.5).
double TextMaskRatio { get; set; } Fraction of text tokens masked by the infilling (0.3).
double TextRandomRatio { get; set; } Fraction of masks replaced by random tokens instead (0.1).
int BosId { get; set; } The beginning-of-sentence token, which the text pre-training wraps around its text (0).
int ConvPositionGroups { get; set; } The convolutional position embedding’s groups (16).
int ConvPositionKernel { get; set; } The convolutional position embedding’s kernel (128).
int EosId { get; set; } The end-of-sentence token, appended to every text (2).
int FeedForwardDim { get; set; } Feed-forward width (3072).
int FinalDim { get; set; } Width of the HuBERT post-net’s projection (256).
int FreezeEncoderUpdates { get; set; } Text-to-speech updates during which the shared encoder stays frozen (20k, the recipe’s freeze_encoder_updates).
int GuidedAttentionHeads { get; set; } Heads of every decoder layer’s cross-attention the guided attention loss applies to (2).
int HopSize { get; set; } Waveform samples per mel frame, paired with the HiFi-GAN upsampling product.
int LatentGroups { get; set; } Codebook groups (2).
int LatentVars { get; set; } Codebook entries per group (100).
int MaskId { get; set; } The mask token of the text infilling (79, <mask>).
int MaskLength { get; set; } The masked spans’ length (10).
int MaxRelativePosition { get; set; } Clipping distance of the encoder’s relative position embedding (160).
int MaxSpeechSamples { get; set; } The longest pre-training recording in samples, randomly cropped to (250000).
int PadId { get; set; } The padding token (1).
int PostnetChannels { get; set; } The post-net’s channels (256).
int PostnetKernel { get; set; } The post-net’s kernel (5).
int PostnetLayers { get; set; } The post-net’s convolutions (5).
int PrenetLayers { get; set; } The speech decoder pre-net’s layers (2).
int PrenetUnits { get; set; } The speech decoder pre-net’s units (256).
int PretrainUpdates { get; set; } Pre-training updates (500k, §3.1); the rate warms up over the first 8% and decays linearly to 0.
int ReductionFactor { get; set; } Mel frames per decoder step (2).
int SamplingSeed { get; set; } Seed of the generation’s pre-net dropout and the x-vector’s pooling noise.
int SpeakerDim { get; set; } Width of the speaker x-vector (512).
int TrainingSeed { get; set; } Seed of the training draws (dropout, masks, crops and the codebook’s Gumbel noise).
int VocoderInitialChannels { get; set; } HiFi-GAN’s first width (512).
int WarmupUpdates { get; set; } Text-to-speech warm-up steps before the inverse-square-root decay (10k, Appendix B.2).
int[] HubertClasses { get; set; } HuBERT label classes per label set (one set of 500 k-means units plus fairseq’s 4 special symbols, 504).
int[] SpeakerEncoderChannels { get; set; } The x-vector’s TDNN channels (512, 512, 512, 512, 1500).
int[] VocoderResBlockKernels { get; set; } HiFi-GAN’s residual block kernels (3, 7, 11).
int[] VocoderUpsampleKernels { get; set; } HiFi-GAN’s transposed-convolution kernels (8, 8, 8, 8).
int[] VocoderUpsampleRates { get; set; } HiFi-GAN’s upsampling rates (4, 4, 4, 4; their product is HopSize).
int[][] VocoderResBlockDilations { get; set; } HiFi-GAN’s residual block dilations (1, 3, 5 each).
string SentencePieceModelPath { get; set; } The character SentencePiece model (spm_char.model), which turning text into tokens needs.

Methods

Method Summary
SpeechT5Options Pretraining() The joint pre-training’s configuration (the recipe’s t5_transformer_base: dropout 0.1, LayerDrop 0.05 in both stacks), training Pretraining.