SpeechGPT<T>
Models & Types · AiDotNet.TextToSpeech.MultiModal
SpeechGPT (Zhang et al., Findings of EMNLP 2023): a large language model with intrinsic cross-modal conversational abilities. Speech becomes discrete mHuBERT units that join LLaMA’s vocabulary, so one model reads and writes both speech and text; a unit vocoder voices the units it writes.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → SpeechGPT<T>
For Beginners
SpeechGPT is a chatbot that has learned speech as if it were another language. It reads “speech units” (symbols a speech model extracts from audio) and writes them back, and a vocoder turns the units it writes into sound.
How It Works
Units (§3.1, §4.1): mHuBERT’s layer-11 features in 1000 k-means clusters, adjacent repeats removed,
written <sosp><u₁>…<eosp> as tokens added to the vocabulary (eq. 1–2).
Tasks are instructions in the release’s template, [Human]: {instruction} <eoh>. [SpeechGPT]:
after the meta instruction: TTS and ASR (§3.1, “{description} This is input: {input}”) and the chain-of-modality
dialogues of App. C ([tq] … ; [ta] … ; [ua] …).
Training (§4.2): stage 1 predicts every token of unlabelled speech’s units (eq. 3); stage 2 fine-tunes on
cross-modal instructions with the loss on the response (eq. 4); stage 3 trains LoRA adapters on chain-of-modality
data. The vocoder is the paper’s multi-speaker unit HiFi-GAN, trained with HiFi-GAN’s discriminators
(TrainVocoder), or the release’s single-speaker one.
Example
var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var options = SpeechGPTOptions.Release7B();
options.Vocoder = SpeechGptVocoder.ReleaseLJ;
var model = new SpeechGPT<float>(architecture, options);
model.LoadReleasedWeights("SpeechGPT-7B-cm");
model.LoadUnitExtractor("mhubert_base_vp_en_es_fr_it3.pt", "mhubert_base_vp_en_es_fr_it3_L11_km1000.bin");
model.LoadVocoder("vocoder/g_00500000");
var audio = model.Synthesize("Today is a sunny day.");
string text = model.Transcribe(recording16kHz);
Constructors
| Constructor | Summary |
|---|---|
SpeechGPT(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
SpeechGPT(NeuralNetworkArchitecture<T> architecture, SpeechGPTOptions options) |
Creates a native, trainable SpeechGPT. |
SpeechGPT(NeuralNetworkArchitecture<T> architecture, string modelPath, SpeechGPTOptions options) |
Creates an ONNX-backed SpeechGPT for inference. |
Properties
| Property | Summary |
|---|---|
bool TrainsOnSampleObjective { get; } |
|
int MaxTextLength { get; } |
The longest sequence the language model takes (2048). |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
SpeechGptResponse<T> Chat(Tensor<T> recording, string text, bool speechResponse) |
A chain-of-modality reply (App. |
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
T TrainChainOfModality(Tensor<T> speechInstruction, string textInstruction, string textResponse, Tensor<T> speechResponse) |
A chain-of-modality step (App. |
T TrainContinuation(Tensor<T> recording) |
Stage 1 (eq. |
T TrainInstruction(string question, string response) |
An instruction step (eq. |
T TrainTranscription(Tensor<T> recording, string transcript) |
A cross-modal ASR step (§3.1): 16 kHz speech and its transcript. |
T TrainVocoder(Tensor<T> recording, int speaker) |
One GAN step of the multi-speaker unit vocoder (§4.1; speech-resynthesis’ train.py) on 16 kHz speech of a speaker: its frame-level units and audio cut to a segment of VocoderSegmentSize samples at a random unit, the discriminators’ leas… |
Tensor<T> Speak(IReadOnlyList<int> units) |
Units voiced by the vocoder (as Speaker with the multi-speaker one). |
Tensor<T> Synthesize(string text) |
|
int[] TokenizeText(string text) |
The token ids of text (no <s>). |
int[] UnitsOf(Tensor<T> recording) |
The units of 16 kHz speech, repeats merged (§3.1). |
string Transcribe(Tensor<T> recording) |
The transcript SpeechGPT writes for 16 kHz speech (its ASR task). |
void LoadReleasedWeights(string directory, string loraDirectory) |
Loads fnlp/SpeechGPT-7B-cm (use Release7B) and, with loraDirectory (fnlp/SpeechGPT-7B-com’s adapter_model.bin, with ChainOfModality), its LoRA adapters; unless one is set, the tokenizer of directory. |
void LoadUnitExtractor(string hubertCheckpoint, string kmeansModel) |
Loads mHuBERT and its k-means model, the unit extractor (mhubert_base_vp_en_es_fr_it3.pt, …_L11_km1000.bin). |
void LoadVocoder(string path) |
Loads the release’s vocoder, fairseq’s single-speaker unit HiFi-GAN (g_00500000); needs ReleaseLJ. |