SpeechGPTOptions
Options & Configuration · AiDotNet.TextToSpeech.MultiModal
Options for SpeechGPT and its speech recognizer.
Inheritance: ModelOptions → TtsModelOptions → SpeechGPTOptions
For Beginners
SpeechGPT teaches a text language model a new “language”: speech units, little symbols a speech model extracts from audio. It can then read units (listen), write units (speak) and translate between them and text. A vocoder turns written units back into sound.
How It Works
The defaults are the paper’s: LLaMA-13B (40 layers, d 5120, 40 heads, SwiGLU 13824) with its vocabulary
expanded by 1000 mHuBERT units and six special tokens (§4.2, eq. 1–2); mHuBERT layer-11 units with repeats merged
(§3.1); the multi-speaker unit vocoder; sampling at temperature 0.8 with top-k 60 and top-p 0.8 over up to 2048
tokens (§5.1); a peak rate of 2e-4 in every stage (App. D). Release7B is the released
fnlp/SpeechGPT-7B-cm’s LLaMA-7B.
Constructors
| Constructor | Summary |
|---|---|
SpeechGPTOptions() |
Creates the paper’s configuration (LLaMA-13B). |
Properties
| Property | Summary |
|---|---|
SpeechGptTrainingStage Stage { get; set; } |
The stage a language-model step follows; cross-modal instruction fine-tuning by default, which TTS and ASR pairs are. |
SpeechGptVocoder Vocoder { get; set; } |
The unit vocoder: the paper’s multi-speaker one by default. |
bool IncludeVocoderDiscriminators { get; set; } |
Whether to build HiFi-GAN’s multi-period and multi-scale discriminators, which training the vocoder needs. |
double LoraAlpha { get; set; } |
|
double LoraDropout { get; set; } |
|
double MaxGradientNorm { get; set; } |
Gradient clipping (Hugging Face Trainer’s default, 1.0). |
double RmsEpsilon { get; set; } |
RMSNorm ε (the release’s 1e-6) and the rotary base (10000). |
double RopeTheta { get; set; } |
|
double Temperature { get; set; } |
Sampling (§5.1): temperature 0.8, top-k 60, top-p 0.8; 0 temperature decodes greedily. |
double TopP { get; set; } |
|
double VocoderLearningRate { get; set; } |
The vocoder’s rate (2e-4, AdamW β 0.8 and 0.99), decayed by 0.999 per epoch of VocoderUpdatesPerEpoch updates; its losses weigh the mel L1 by 45 and the duration loss by 1. |
double VocoderLearningRateDecay { get; set; } |
|
double WarmupFraction { get; set; } |
|
int HubertConvChannels { get; set; } |
|
int HubertHeads { get; set; } |
|
int HubertIntermediate { get; set; } |
|
int HubertLayer { get; set; } |
|
int HubertWidth { get; set; } |
mHuBERT’s width (768), the layer read (11), heads (12), feed-forward width (3072) and feature-encoder channels (512). |
int Intermediate { get; set; } |
|
int Layers { get; set; } |
LLaMA layers (40) and SwiGLU width (13824). |
int LoraRank { get; set; } |
Stage 3’s LoRA on the query and value projections: rank 8, alpha 16, dropout 0.05 (App. |
int MaxNewTokens { get; set; } |
The most tokens a response may have, and the fewest before it may end (the release’s 10). |
int MinNewTokens { get; set; } |
|
int Speaker { get; set; } |
|
int TopK { get; set; } |
|
int TotalSteps { get; set; } |
Steps of the cosine schedule (App. |
int Units { get; set; } |
Discrete units, K (1000 mHuBERT clusters). |
int VocoderChannels { get; set; } |
|
int VocoderEmbeddingDim { get; set; } |
The vocoder’s embedding width (128), first width (512), upsampling (5·4·4·2·2 = 320 samples per unit, 50 Hz) and kernels. |
int VocoderSegmentSize { get; set; } |
The vocoder’s training segment in samples (speech-resynthesis’ 8960, 28 units). |
int VocoderSpeakers { get; set; } |
The multi-speaker vocoder’s speaker table (speech-resynthesis’ 200) and the speaker synthesis uses. |
int VocoderUpdatesPerEpoch { get; set; } |
|
int[] VocoderUpsampleKernels { get; set; } |
|
int[] VocoderUpsampleRates { get; set; } |
|
string AsrInstruction { get; set; } |
|
string MetaInstruction { get; set; } |
The release’s meta instruction, before every prompt. |
string TokenizerDirectory { get; set; } |
A directory with LLaMA’s tokenizer.model and the release’s added_tokens.json; text needs it. |
string TtsInstruction { get; set; } |
The task descriptions of TTS and ASR (App. |
Methods
| Method | Summary |
|---|---|
SpeechGPTOptions Release7B() |
The release’s LLaMA-7B (fnlp/SpeechGPT-7B-cm): 32 layers, d 4096, 32 heads, SwiGLU 11008. |