SpeechGPT<T>

Models & Types · AiDotNet.TextToSpeech.MultiModal

SpeechGPT (Zhang et al., Findings of EMNLP 2023): a large language model with intrinsic cross-modal conversational abilities. Speech becomes discrete mHuBERT units that join LLaMA’s vocabulary, so one model reads and writes both speech and text; a unit vocoder voices the units it writes.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → SpeechGPT<T>

For Beginners

SpeechGPT is a chatbot that has learned speech as if it were another language. It reads “speech units” (symbols a speech model extracts from audio) and writes them back, and a vocoder turns the units it writes into sound.

How It Works

Units (§3.1, §4.1): mHuBERT’s layer-11 features in 1000 k-means clusters, adjacent repeats removed, written <sosp><u₁>…<eosp> as tokens added to the vocabulary (eq. 1–2).

Tasks are instructions in the release’s template, [Human]: {instruction} <eoh>. [SpeechGPT]: after the meta instruction: TTS and ASR (§3.1, “{description} This is input: {input}”) and the chain-of-modality dialogues of App. C ([tq] … ; [ta] … ; [ua] …).

Training (§4.2): stage 1 predicts every token of unlabelled speech’s units (eq. 3); stage 2 fine-tunes on cross-modal instructions with the loss on the response (eq. 4); stage 3 trains LoRA adapters on chain-of-modality data. The vocoder is the paper’s multi-speaker unit HiFi-GAN, trained with HiFi-GAN’s discriminators (TrainVocoder), or the release’s single-speaker one.

Example

var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var options = SpeechGPTOptions.Release7B();
options.Vocoder = SpeechGptVocoder.ReleaseLJ;
var model = new SpeechGPT<float>(architecture, options);
model.LoadReleasedWeights("SpeechGPT-7B-cm");
model.LoadUnitExtractor("mhubert_base_vp_en_es_fr_it3.pt", "mhubert_base_vp_en_es_fr_it3_L11_km1000.bin");
model.LoadVocoder("vocoder/g_00500000");
var audio = model.Synthesize("Today is a sunny day.");
string text = model.Transcribe(recording16kHz);

Constructors

Constructor Summary
SpeechGPT(NeuralNetworkArchitecture<T> architecture) Calls the fully specified overload with the omitted parameters’ defaults.
SpeechGPT(NeuralNetworkArchitecture<T> architecture, SpeechGPTOptions options) Creates a native, trainable SpeechGPT.
SpeechGPT(NeuralNetworkArchitecture<T> architecture, string modelPath, SpeechGPTOptions options) Creates an ONNX-backed SpeechGPT for inference.

Properties

Property Summary
bool TrainsOnSampleObjective { get; }
int MaxTextLength { get; } The longest sequence the language model takes (2048).

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
SpeechGptResponse<T> Chat(Tensor<T> recording, string text, bool speechResponse) A chain-of-modality reply (App.
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
T TrainChainOfModality(Tensor<T> speechInstruction, string textInstruction, string textResponse, Tensor<T> speechResponse) A chain-of-modality step (App.
T TrainContinuation(Tensor<T> recording) Stage 1 (eq.
T TrainInstruction(string question, string response) An instruction step (eq.
T TrainTranscription(Tensor<T> recording, string transcript) A cross-modal ASR step (§3.1): 16 kHz speech and its transcript.
T TrainVocoder(Tensor<T> recording, int speaker) One GAN step of the multi-speaker unit vocoder (§4.1; speech-resynthesis’ train.py) on 16 kHz speech of a speaker: its frame-level units and audio cut to a segment of VocoderSegmentSize samples at a random unit, the discriminators’ leas…
Tensor<T> Speak(IReadOnlyList<int> units) Units voiced by the vocoder (as Speaker with the multi-speaker one).
Tensor<T> Synthesize(string text)
int[] TokenizeText(string text) The token ids of text (no <s>).
int[] UnitsOf(Tensor<T> recording) The units of 16 kHz speech, repeats merged (§3.1).
string Transcribe(Tensor<T> recording) The transcript SpeechGPT writes for 16 kHz speech (its ASR task).
void LoadReleasedWeights(string directory, string loraDirectory) Loads fnlp/SpeechGPT-7B-cm (use Release7B) and, with loraDirectory (fnlp/SpeechGPT-7B-com’s adapter_model.bin, with ChainOfModality), its LoRA adapters; unless one is set, the tokenizer of directory.
void LoadUnitExtractor(string hubertCheckpoint, string kmeansModel) Loads mHuBERT and its k-means model, the unit extractor (mhubert_base_vp_en_es_fr_it3.pt, …_L11_km1000.bin).
void LoadVocoder(string path) Loads the release’s vocoder, fairseq’s single-speaker unit HiFi-GAN (g_00500000); needs ReleaseLJ.