SpeechT5Options
Options & Configuration · AiDotNet.TextToSpeech.MultiModal
Options for SpeechT5.
Inheritance: ModelOptions → TtsModelOptions → SpeechT5Options
For Beginners
SpeechT5 is one encoder-decoder pre-trained on both speech and text and fine-tuned to read text aloud in the voice of a speaker embedding (an x-vector computed from a recording). These settings size the networks and control training and generation.
How It Works
The defaults are the paper’s base model (Ao et al. 2022: 12 encoder and 6 decoder layers of width 768, 12
heads, feed-forward 3072, relative position embeddings; 100 × 2 codebook entries mixed into 10% of the states) with
the official recipe filling what the paper leaves out (t5_transformer_base, the pre-training and TTS
commands), and the paper’s training rates: pre-training warms up over 8% of 500k updates to 2e-4 then decays
linearly; text-to-speech fine-tuning warms up over 10k of 120k steps to 4e-4 then decays by the inverse square root.
The released weights (microsoft/speecht5_tts) use these sizes.
Constructors
| Constructor | Summary |
|---|---|
SpeechT5Options() |
Creates the paper’s base configuration. |
Properties
| Property | Summary |
|---|---|
SpeechT5Stage CurrentStage { get; set; } |
The objective SpeechT5 trains (text-to-speech fine-tuning). |
SpeechT5StopCriterion StopCriterion { get; set; } |
When generation stops (the official code’s: any frame). |
ValueTuple<int, int, int>[] FeatureLayers { get; set; } |
The speech encoder pre-net’s convolutions as (channels, kernel, stride): wav2vec 2.0’s seven. |
bool BertInitialization { get; set; } |
Whether a fresh model draws its weights as the official pre-training’s --bert-init does (N(0, 0.02) linear layers and embeddings, zero biases; true). |
double ActivationDropout { get; set; } |
Dropout after the feed-forward activation (0.15, the TTS recipe’s). |
double AttentionDropout { get; set; } |
Attention-weight dropout (0.15, the TTS recipe’s). |
double CodebookProbability { get; set; } |
Fraction of encoder states replaced by their quantized vectors (0.1). |
double DecoderLayerDrop { get; set; } |
LayerDrop of the decoder layers (0 in the TTS recipe; 0.05 in pre-training). |
double DiversityWeight { get; set; } |
Weight of the codebook diversity loss (0.1). |
double EncoderLayerDrop { get; set; } |
LayerDrop of the encoder layers (0 in the TTS recipe; 0.05 in pre-training). |
double FeaturePenaltyWeight { get; set; } |
Weight of the feature penalty (10). |
double GuidedAttentionSigma { get; set; } |
The guided attention loss’s σ (0.4). |
double LabelRate { get; set; } |
HuBERT labels per second (50). |
double MaskProbability { get; set; } |
The span masking’s start probability (0.8 → ≈8% of frames start a span, §2.3). |
double MaxGradientNorm { get; set; } |
Text-to-speech’s gradient-norm clip (25, the recipe’s). |
double MaxLengthRatio { get; set; } |
The longest generation, in decoder steps per text token times r (20). |
double OptimizerBeta1 { get; set; } |
Adam β1 (0.9). |
double OptimizerBeta2 { get; set; } |
Adam β2 (0.98). |
double OptimizerEpsilon { get; set; } |
Text-to-speech’s Adam ε (1e-8, the recipe’s default). |
double PoissonLambda { get; set; } |
The Poisson λ of the infilling’s span lengths (3.5). |
double PositionalDropout { get; set; } |
Dropout after the scaled positional encodings (0.1). |
double PostnetDropout { get; set; } |
The post-net’s dropout (0.5). |
double PrenetDropout { get; set; } |
The speech decoder pre-net’s always-on dropout (0.5, Tacotron 2’s). |
double PretrainEpsilon { get; set; } |
Pre-training’s Adam ε (1e-6, the recipe’s). |
double PretrainLearningRate { get; set; } |
Pre-training’s peak learning rate (2e-4, §3.1). |
double PretrainMaxGradientNorm { get; set; } |
Pre-training’s gradient-norm clip (5, the recipe’s). |
double PretrainWarmupFraction { get; set; } |
Fraction of pre-training updates spent warming up (0.08). |
double PretrainWeightDecay { get; set; } |
Pre-training’s decoupled weight decay (0.01, the recipe’s). |
double StopPositiveWeight { get; set; } |
Weight of the stop token’s positive class in its BCE (5). |
double StopThreshold { get; set; } |
Stop probability that ends generation (0.5). |
double TextMaskRatio { get; set; } |
Fraction of text tokens masked by the infilling (0.3). |
double TextRandomRatio { get; set; } |
Fraction of masks replaced by random tokens instead (0.1). |
int BosId { get; set; } |
The beginning-of-sentence token, which the text pre-training wraps around its text (0). |
int ConvPositionGroups { get; set; } |
The convolutional position embedding’s groups (16). |
int ConvPositionKernel { get; set; } |
The convolutional position embedding’s kernel (128). |
int EosId { get; set; } |
The end-of-sentence token, appended to every text (2). |
int FeedForwardDim { get; set; } |
Feed-forward width (3072). |
int FinalDim { get; set; } |
Width of the HuBERT post-net’s projection (256). |
int FreezeEncoderUpdates { get; set; } |
Text-to-speech updates during which the shared encoder stays frozen (20k, the recipe’s freeze_encoder_updates). |
int GuidedAttentionHeads { get; set; } |
Heads of every decoder layer’s cross-attention the guided attention loss applies to (2). |
int HopSize { get; set; } |
Waveform samples per mel frame, paired with the HiFi-GAN upsampling product. |
int LatentGroups { get; set; } |
Codebook groups (2). |
int LatentVars { get; set; } |
Codebook entries per group (100). |
int MaskId { get; set; } |
The mask token of the text infilling (79, <mask>). |
int MaskLength { get; set; } |
The masked spans’ length (10). |
int MaxRelativePosition { get; set; } |
Clipping distance of the encoder’s relative position embedding (160). |
int MaxSpeechSamples { get; set; } |
The longest pre-training recording in samples, randomly cropped to (250000). |
int PadId { get; set; } |
The padding token (1). |
int PostnetChannels { get; set; } |
The post-net’s channels (256). |
int PostnetKernel { get; set; } |
The post-net’s kernel (5). |
int PostnetLayers { get; set; } |
The post-net’s convolutions (5). |
int PrenetLayers { get; set; } |
The speech decoder pre-net’s layers (2). |
int PrenetUnits { get; set; } |
The speech decoder pre-net’s units (256). |
int PretrainUpdates { get; set; } |
Pre-training updates (500k, §3.1); the rate warms up over the first 8% and decays linearly to 0. |
int ReductionFactor { get; set; } |
Mel frames per decoder step (2). |
int SamplingSeed { get; set; } |
Seed of the generation’s pre-net dropout and the x-vector’s pooling noise. |
int SpeakerDim { get; set; } |
Width of the speaker x-vector (512). |
int TrainingSeed { get; set; } |
Seed of the training draws (dropout, masks, crops and the codebook’s Gumbel noise). |
int VocoderInitialChannels { get; set; } |
HiFi-GAN’s first width (512). |
int WarmupUpdates { get; set; } |
Text-to-speech warm-up steps before the inverse-square-root decay (10k, Appendix B.2). |
int[] HubertClasses { get; set; } |
HuBERT label classes per label set (one set of 500 k-means units plus fairseq’s 4 special symbols, 504). |
int[] SpeakerEncoderChannels { get; set; } |
The x-vector’s TDNN channels (512, 512, 512, 512, 1500). |
int[] VocoderResBlockKernels { get; set; } |
HiFi-GAN’s residual block kernels (3, 7, 11). |
int[] VocoderUpsampleKernels { get; set; } |
HiFi-GAN’s transposed-convolution kernels (8, 8, 8, 8). |
int[] VocoderUpsampleRates { get; set; } |
HiFi-GAN’s upsampling rates (4, 4, 4, 4; their product is HopSize). |
int[][] VocoderResBlockDilations { get; set; } |
HiFi-GAN’s residual block dilations (1, 3, 5 each). |
string SentencePieceModelPath { get; set; } |
The character SentencePiece model (spm_char.model), which turning text into tokens needs. |
Methods
| Method | Summary |
|---|---|
SpeechT5Options Pretraining() |
The joint pre-training’s configuration (the recipe’s t5_transformer_base: dropout 0.1, LayerDrop 0.05 in both stacks), training Pretraining. |