GLM4VoiceOptions

Options & Configuration · AiDotNet.TextToSpeech.MultiModal

Options for GLM4Voice.

Inheritance: ModelOptions → TtsModelOptions → GLM4VoiceOptions

For Beginners

GLM-4-Voice is a voice chatbot. It turns speech into “speech words” (tokens), a large language model writes a reply that mixes text words and speech words, and a decoder turns the speech words back into sound. These settings size each of those parts and control how replies are generated and how each part is trained.

How It Works

The defaults are the release (THUDM/glm-4-voice-9b, glm-4-voice-tokenizer, glm-4-voice-decoder):

  • the ChatGLM language model: 40 layers, d 4096, 32 heads of 128 in 2 key-value groups, SwiGLU 13696 and a

168960-token vocabulary of text and speech tokens;

  • the Whisper-VQ speech tokenizer: whisper-large-v3’s encoder up to layer 16, pooled 4× and quantized to 16384

codes at 12.5 Hz;

  • the CosyVoice flow decoder: a 6-block Conformer with 10-token attention blocks and a conditional flow-matching

U-Net, then HiFT at 22.05 kHz.

Constructors

Constructor Summary
GLM4VoiceOptions() Creates the release’s configuration.

Properties

Property Summary
CosyVoiceFlowLoss FlowLoss { get; set; } The flow-matching loss: CosyVoice’s paper’s absolute error (the release’s config also names l1), or its code’s squared error.
GlmVoiceDecoding Decoding { get; set; } How speech tokens are decoded to audio: the paper’s 0.8 s blocks by default.
GlmVoiceGeneration Generation { get; set; } The reply’s layout: streaming thoughts by default.
GlmVoiceTrainingStage Stage { get; set; } Which stage the language model’s training steps follow: fine-tuning by default.
GlmVoiceTtsFormat TtsFormat { get; set; } What Synthesize does with a text: forced-text thoughts by default.
bool IncludeSpeechTokenizerTraining { get; set; } Whether to build the speech tokenizer’s training stack: whisper-large-v3’s remaining encoder layers and its decoder, which carry the ASR loss that trains the tokenizer (§3.1).
bool StreamingTruncation { get; set; } Whether decoder training uses the paper’s truncated streaming samples (§3.2: the first n·b seconds of each utterance), as its fine-tuning stage does.
double CommitCoefficient { get; set; } The commitment coefficient (0.25) and the quantize loss’s scale (10): the release’s quantize_commit_coefficient and quantize_loss_scale.
double DecoderLearningRate { get; set; } The decoder’s training rate (Zeng et al.
double EmaDecay { get; set; } The codebook’s EMA decay (0.99).
double FineTuningDropout { get; set; } Dropout on the hidden states during fine-tuning (§4.2.2: 0.5).
double InferenceCfgRate { get; set; } The classifier-free guidance weight at inference (0.7) and the condition-drop rate in training (0.2).
double MaxGradientNorm { get; set; } Gradient clipping (§4.2.2: 1.0).
double NormEpsilon { get; set; } The RMSNorm ε (3.90625e-8).
double PreTrainingLearningRate { get; set; } Pre-training’s peak rate (§4.1.1: 6e-5); fine-tuning’s is LearningRate (§4.2.2: 1e-5).
double QuantizeLossScale { get; set; }
double RopeRatio { get; set; } The rotary base multiplier (rope_ratio, 1).
double SpeechTokenizerLearningRate { get; set; } The speech tokenizer’s fine-tuning rate (Zeng et al.
double Temperature { get; set; } Sampling temperature (the web demo’s 0.2) and nucleus (0.8); the release always samples.
double TopP { get; set; }
double TrainingCfgRate { get; set; }
int AsrDecoderLayers { get; set; } Whisper decoder layers (32), vocabulary (51866) and positions (448), for the ASR loss.
int AsrDecoderStartId { get; set; } Whisper’s decoder_start_token_id (<|startoftranscript|>, 50258), which the ASR decoder’s input starts with.
int AsrMaxTargetPositions { get; set; }
int AsrVocabulary { get; set; }
int CausalBlockSize { get; set; } Encoder frames per block-causal attention block before pooling (200: 4 s).
int DecaySteps { get; set; } Steps over which the rate decays linearly to a tenth (§4.1.1, §4.2.2 give the endpoints, not the length).
int DecoderBlocks { get; set; }
int DecoderHeadDim { get; set; }
int DecoderHeads { get; set; }
int DecoderMidBlocks { get; set; }
int FeedForwardDim { get; set; } The SwiGLU inner width (13696).
int FlowBlockTokens { get; set; } Speech tokens per decoding block and Conformer attention block (10: b = 0.8 s, §3.2).
int FlowDim { get; set; } The flow’s token embedding and Conformer width (512).
int FlowFeedForwardDim { get; set; }
int FlowHeads { get; set; } The Conformer’s heads (8), feed-forward width (2048) and blocks (6).
int FlowLayers { get; set; }
int FlowTimesteps { get; set; } Euler steps of the flow (10).
int HeadDim { get; set; } Width of each attention head (kv_channels, 128).
int HiftBaseChannels { get; set; } HiFT’s base channels (512) and F0 predictor channels (512).
int HiftF0Channels { get; set; }
int HopSize { get; set; } The released HiFT decoder’s fixed hop, 256 samples.
int KeyValueGroups { get; set; } Key-value groups of the multi-query attention (2).
int Layers { get; set; } Transformer layers of the language model (40).
int MaxNewTokens { get; set; } The most tokens a reply may have (the web demo’s 2000).
int MelChannels { get; set; } The released speech decoder’s fixed 80-band mel representation.
int NumDecoderLayers { get; set; } Gets or sets the language-model decoder depth, forwarding to Layers.
int NumEncoderLayers { get; set; } Gets or sets the speech-tokenizer encoder depth, forwarding to SpeechTokenizerLayers.
int PoolingKernel { get; set; } The average pooling’s kernel and stride (4).
int PoolingPosition { get; set; } The layer after which the pooling runs (16).
int QuantizePosition { get; set; } The layer after which the vector quantizer runs (16).
int RestartInterval { get; set; } Codebook updates between dead-code restarts (100).
int SampleRate { get; set; } The released HiFT decoder’s fixed sample rate, 22050 Hz.
int SpeechCodebookSize { get; set; } Speech tokens (VQ codes): 16384.
int SpeechEpochs { get; set; } Fine-tuning epochs of the speech part and of the text part of each split sample (§4.2.2: 20 and 4).
int SpeechTokenizerDim { get; set; } The speech tokenizer’s width (whisper-large-v3: 1280).
int SpeechTokenizerFeedForwardDim { get; set; } The speech tokenizer’s feed-forward width (5120).
int SpeechTokenizerHeads { get; set; } The speech tokenizer’s attention heads (20).
int SpeechTokenizerLayers { get; set; } Whisper encoder layers in all (32); tokenizing runs only the first QuantizePosition.
int SpeechTokenizerMaxPositions { get; set; } The encoder’s position table (1500 frames, 30 s).
int SpeechTokensPerChunk { get; set; }
int TextEpochs { get; set; }
int TextTokensPerChunk { get; set; } Text tokens, then speech tokens, per streaming-thoughts chunk (13 and 26).
int[] DecoderChannels { get; set; } The U-Net estimator’s channels ([256, 256]), heads (8 of 64), transformer blocks per level (4) and middle blocks (12).
string SystemPrompt { get; set; } The system prompt; null uses the release’s (the web demo’s for a speech or a text input under streaming thoughts, Zeng et al.
string TokenizerDirectory { get; set; } A model directory with GLM-4’s tokenizer.model and tokenizer_config.json; text needs it.