SpeechGPTOptions

Options & Configuration · AiDotNet.TextToSpeech.MultiModal

Options for SpeechGPT and its speech recognizer.

Inheritance: ModelOptions → TtsModelOptions → SpeechGPTOptions

For Beginners

SpeechGPT teaches a text language model a new “language”: speech units, little symbols a speech model extracts from audio. It can then read units (listen), write units (speak) and translate between them and text. A vocoder turns written units back into sound.

How It Works

The defaults are the paper’s: LLaMA-13B (40 layers, d 5120, 40 heads, SwiGLU 13824) with its vocabulary expanded by 1000 mHuBERT units and six special tokens (§4.2, eq. 1–2); mHuBERT layer-11 units with repeats merged (§3.1); the multi-speaker unit vocoder; sampling at temperature 0.8 with top-k 60 and top-p 0.8 over up to 2048 tokens (§5.1); a peak rate of 2e-4 in every stage (App. D). Release7B is the released fnlp/SpeechGPT-7B-cm’s LLaMA-7B.

Constructors

Constructor Summary
SpeechGPTOptions() Creates the paper’s configuration (LLaMA-13B).

Properties

Property Summary
SpeechGptTrainingStage Stage { get; set; } The stage a language-model step follows; cross-modal instruction fine-tuning by default, which TTS and ASR pairs are.
SpeechGptVocoder Vocoder { get; set; } The unit vocoder: the paper’s multi-speaker one by default.
bool IncludeVocoderDiscriminators { get; set; } Whether to build HiFi-GAN’s multi-period and multi-scale discriminators, which training the vocoder needs.
double LoraAlpha { get; set; }
double LoraDropout { get; set; }
double MaxGradientNorm { get; set; } Gradient clipping (Hugging Face Trainer’s default, 1.0).
double RmsEpsilon { get; set; } RMSNorm ε (the release’s 1e-6) and the rotary base (10000).
double RopeTheta { get; set; }
double Temperature { get; set; } Sampling (§5.1): temperature 0.8, top-k 60, top-p 0.8; 0 temperature decodes greedily.
double TopP { get; set; }
double VocoderLearningRate { get; set; } The vocoder’s rate (2e-4, AdamW β 0.8 and 0.99), decayed by 0.999 per epoch of VocoderUpdatesPerEpoch updates; its losses weigh the mel L1 by 45 and the duration loss by 1.
double VocoderLearningRateDecay { get; set; }
double WarmupFraction { get; set; }
int HubertConvChannels { get; set; }
int HubertHeads { get; set; }
int HubertIntermediate { get; set; }
int HubertLayer { get; set; }
int HubertWidth { get; set; } mHuBERT’s width (768), the layer read (11), heads (12), feed-forward width (3072) and feature-encoder channels (512).
int Intermediate { get; set; }
int Layers { get; set; } LLaMA layers (40) and SwiGLU width (13824).
int LoraRank { get; set; } Stage 3’s LoRA on the query and value projections: rank 8, alpha 16, dropout 0.05 (App.
int MaxNewTokens { get; set; } The most tokens a response may have, and the fewest before it may end (the release’s 10).
int MinNewTokens { get; set; }
int Speaker { get; set; }
int TopK { get; set; }
int TotalSteps { get; set; } Steps of the cosine schedule (App.
int Units { get; set; } Discrete units, K (1000 mHuBERT clusters).
int VocoderChannels { get; set; }
int VocoderEmbeddingDim { get; set; } The vocoder’s embedding width (128), first width (512), upsampling (5·4·4·2·2 = 320 samples per unit, 50 Hz) and kernels.
int VocoderSegmentSize { get; set; } The vocoder’s training segment in samples (speech-resynthesis’ 8960, 28 units).
int VocoderSpeakers { get; set; } The multi-speaker vocoder’s speaker table (speech-resynthesis’ 200) and the speaker synthesis uses.
int VocoderUpdatesPerEpoch { get; set; }
int[] VocoderUpsampleKernels { get; set; }
int[] VocoderUpsampleRates { get; set; }
string AsrInstruction { get; set; }
string MetaInstruction { get; set; } The release’s meta instruction, before every prompt.
string TokenizerDirectory { get; set; } A directory with LLaMA’s tokenizer.model and the release’s added_tokens.json; text needs it.
string TtsInstruction { get; set; } The task descriptions of TTS and ASR (App.

Methods

Method Summary
SpeechGPTOptions Release7B() The release’s LLaMA-7B (fnlp/SpeechGPT-7B-cm): 32 layers, d 4096, 32 heads, SwiGLU 11008.