SpeechGPT<T>

Models & Types · AiDotNet.SpeechRecognition.LLMIntegrated

SpeechGPT’s speech recognition (Zhang et al. 2023): the same LLaMA over text and unit tokens as SpeechGPT, given its ASR instruction and a recording’s mHuBERT units, writing the transcript.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → SpeechGPT<T>

Implements: ISpeechRecognizer<T>

For Beginners

SpeechGPT is a language model that has learned speech units as words. To transcribe, it turns speech into units and writes the text that answers “transcribe this”.

How It Works

The recording becomes merged mHuBERT layer-11 units, written <sosp><u>…<eosp> (§3.1); the question is {ASR description} This is input: {units} in the release’s template, and the model continues with the transcript. Training is stage 2’s cross-modal instruction fine-tuning, the loss on the transcript (eq. 4), AdamW at 2e-4 with a cosine schedule (App. D and the release’s scripts).

Example

var options = SpeechGPTOptions.Release7B();
var model = new SpeechGPT<float>(new NeuralNetworkArchitecture<float>(InputType.OneDimensional,
    NeuralNetworkTaskType.SpeechRecognition, inputSize: 1, outputSize: 1), options);
model.LoadReleasedWeights("SpeechGPT-7B-cm");
model.LoadUnitExtractor("mhubert_base_vp_en_es_fr_it3.pt", "mhubert_base_vp_en_es_fr_it3_L11_km1000.bin");
var result = model.Transcribe(recording);

Constructors

Constructor Summary
SpeechGPT(NeuralNetworkArchitecture<T> architecture, SpeechGPTOptions options) Creates SpeechGPT’s speech recognizer.

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
TrainingObjectiveKind TrainingObjectiveKind { get; }
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
LayerInputDomain GetOutputDomain(int[] outputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(Tensor<T> input, Tensor<T> target)
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ResolveTrainingTarget(Tensor<T> input, Tensor<T> proposedTarget)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps)
int[] TokenizeText(string text) The token ids of text (no <s>).
string DetectLanguage(Tensor<T> audio)
void LoadReleasedWeights(string directory) Loads fnlp/SpeechGPT-7B-cm (use Release7B) and, unless one is set, its tokenizer.
void LoadUnitExtractor(string hubertCheckpoint, string kmeansModel) Loads mHuBERT and its k-means model, the unit extractor.