GLM4Voice<T>

Models & Types · AiDotNet.TextToSpeech.MultiModal

GLM-4-Voice (Zeng et al. 2024): an end-to-end spoken chatbot. A 12.5 Hz supervised speech tokenizer turns speech into tokens; GLM-4-9B, extended by those tokens and pre-trained on a trillion speech-text tokens, replies in text and speech tokens interleaved; a flow-matching decoder streams the speech tokens to audio.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → GLM4Voice<T>

For Beginners

GLM-4-Voice listens and talks. Your speech becomes “speech words”, a large language model writes its answer as a mix of normal words and speech words, and a decoder turns the speech words into sound.

How It Works

Speech tokenizer (§3.1): whisper-large-v3’s encoder with causal convolutions and block-causal attention, average-pooled 4× and vector-quantized (16384 codes) after its 16th layer; fine-tuned on ASR so the codes keep the semantics.

Language model (§4): ChatGLM’s decoder (RMSNorm, multi-query attention with rotary positions on half of each head, SwiGLU), its vocabulary grown by the speech tokens. Replies use “streaming thoughts” (§3.3): 13 text tokens, then 26 speech tokens, alternating, so speech can start before the text is finished.

Speech decoder (§3.2): CosyVoice’s flow matching with a Conformer that attends in 10-token blocks, decoding 0.8 s blocks with what came before as the prompt, then HiFT.

Synthesize reads a text aloud (by default the paper’s speech-and-text-to-speech subtask, the text fed as the reply’s text); Chat and Chat answer a text or a 16 kHz recording.

Example

var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new GLM4Voice<float>(architecture, new GLM4VoiceOptions { TokenizerDirectory = "glm-4-voice-9b" });
model.LoadLanguageModel("glm-4-voice-9b");
model.LoadSpeechTokenizer("glm-4-voice-tokenizer");
model.LoadSpeechDecoder("glm-4-voice-decoder");
var audio = model.Synthesize("Hello world.");
var reply = model.Chat(recording16kHz);           // reply.Text, reply.Audio

Constructors

Constructor Summary
GLM4Voice(NeuralNetworkArchitecture<T> architecture) Calls the fully specified overload with the omitted parameters’ defaults.
GLM4Voice(NeuralNetworkArchitecture<T> architecture, GLM4VoiceOptions options) Creates a native, trainable GLM-4-Voice.
GLM4Voice(NeuralNetworkArchitecture<T> architecture, string modelPath, GLM4VoiceOptions options) Creates an ONNX-backed GLM-4-Voice for inference.

Properties

Property Summary
bool TrainsOnSampleObjective { get; }
int AudioOffset { get; } The id of <|audio_0|>: speech token c is this plus c.
int MaxTextLength { get; } The longest token sequence the language model takes (8192).

Methods

Method Summary
GlmVoiceReply<T> Chat(Tensor<T> recording) GLM-4-Voice’s reply to 16 kHz speech (the web demo’s audio input).
GlmVoiceReply<T> Chat(string text) GLM-4-Voice’s reply to 16 kHz speech (the web demo’s audio input).
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
T TrainLanguageModel(int[] ids, bool[] targets) One language-model update of any sequence: targets marks the tokens the loss predicts (all of them for pre-training’s text, speech and interleaved data; the target part of a supervised pair, §4.1).
T TrainSpeechDecoder(IReadOnlyList<int> speechTokens, Tensor<T> audio) One update of the speech decoder’s flow (§3.2; Zeng et al.
T TrainSpeechTokenizer(Tensor<T> recording, IReadOnlyList<int> labels) One update of the speech tokenizer (§3.1; Zeng et al.
Tensor<T> DecodeSpeech(IReadOnlyList<int> speechTokens) Speech tokens (codes) to a 22.05 kHz waveform with the speech decoder (Decoding).
Tensor<T> MelSpectrogram(Tensor<T> audio) The decoder’s mel [1, 80, frames] of 22.05 kHz audio (CosyVoice’s mel_spectrogram: FFT 1024, hop 256, 0 – 8 kHz, not centred).
Tensor<T> Synthesize(string text)
double TrainDialogue(GlmVoiceDialogueTurn turn, int epoch) The fine-tuning of one turn in epoch epoch (§4.2.2): the turn split in two, the text reply’s loss with the speech masked while epoch < TextEpochs, and the speech reply’s loss with the text masked while it is below SpeechEpochs.
int[] SpeechTokensOf(Tensor<T> recording) The speech tokens of 16 kHz audio (extract_speech_token: 30-second chunks, the valid frames’ codes).
int[] TokenizeText(string text) The text token ids of text, without the [gMASK]<sop> prefix.
string DecodeText(IEnumerable<int> ids) The text of token ids; speech tokens are dropped.
void LoadLanguageModel(string directory) Loads THUDM/glm-4-voice-9b (its safetensors shards) into the language model and, unless one is set, its tokenizer.
void LoadSpeechDecoder(string directory) Loads THUDM/glm-4-voice-decoder (flow.pt and hift.pt).
void LoadSpeechTokenizer(string directory) Loads THUDM/glm-4-voice-tokenizer (model.safetensors: the encoder up to the quantizer, the codebook and its EMA state).