SpeechGPT<T>
Models & Types · AiDotNet.SpeechRecognition.LLMIntegrated
SpeechGPT’s speech recognition (Zhang et al. 2023): the same LLaMA over text and unit tokens as
SpeechGPT, given its ASR instruction and a recording’s mHuBERT units, writing the transcript.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → SpeechGPT<T>
Implements: ISpeechRecognizer<T>
For Beginners
SpeechGPT is a language model that has learned speech units as words. To transcribe, it turns speech into units and writes the text that answers “transcribe this”.
How It Works
The recording becomes merged mHuBERT layer-11 units, written <sosp><u>…<eosp>
(§3.1); the question is {ASR description} This is input: {units} in the release’s template, and the model
continues with the transcript. Training is stage 2’s cross-modal instruction fine-tuning, the loss on the
transcript (eq. 4), AdamW at 2e-4 with a cosine schedule (App. D and the release’s scripts).
Example
var options = SpeechGPTOptions.Release7B();
var model = new SpeechGPT<float>(new NeuralNetworkArchitecture<float>(InputType.OneDimensional,
NeuralNetworkTaskType.SpeechRecognition, inputSize: 1, outputSize: 1), options);
model.LoadReleasedWeights("SpeechGPT-7B-cm");
model.LoadUnitExtractor("mhubert_base_vp_en_es_fr_it3.pt", "mhubert_base_vp_en_es_fr_it3_L11_km1000.bin");
var result = model.Transcribe(recording);
Constructors
| Constructor | Summary |
|---|---|
SpeechGPT(NeuralNetworkArchitecture<T> architecture, SpeechGPTOptions options) |
Creates SpeechGPT’s speech recognizer. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
TrainingObjectiveKind TrainingObjectiveKind { get; } |
|
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
|
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
LayerInputDomain GetOutputDomain(int[] outputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(Tensor<T> input, Tensor<T> target) |
|
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
Tensor<T> ResolveTrainingTarget(Tensor<T> input, Tensor<T> proposedTarget) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
|
int[] TokenizeText(string text) |
The token ids of text (no <s>). |
string DetectLanguage(Tensor<T> audio) |
|
void LoadReleasedWeights(string directory) |
Loads fnlp/SpeechGPT-7B-cm (use Release7B) and, unless one is set, its tokenizer. |
void LoadUnitExtractor(string hubertCheckpoint, string kmeansModel) |
Loads mHuBERT and its k-means model, the unit extractor. |