WhisperSpeech<T>

Models & Types · AiDotNet.TextToSpeech.MultiModal

WhisperSpeech (Collabora 2024): text-to-speech by inverting Whisper, after SPEAR-TTS (Kharitonov et al. 2023). A text-to-semantic transformer writes the semantic tokens Whisper’s quantized encoder would hear for the text, a semantic-to-acoustic transformer writes EnCodec’s tokens for them in a speaker’s voice (an ECAPA-TDNN embedding), and Vocos turns the EnCodec tokens into 24 kHz speech.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → WhisperSpeech<T>

Implements: ICodecTts<T>

For Beginners

WhisperSpeech first decides “what is said” as the tokens a speech recognizer would hear, then “how it sounds” as audio-codec tokens in a chosen voice, and finally turns those into a waveform.

How It Works

Built as the release (collabora/WhisperSpeech) runs it; see WhisperSpeechOptions and docs/text-to-speech/paper-fidelity-decisions.md. Text goes in as UTF-8 bytes; speech comes out at 24 kHz. Without a SpeakerEmbedding the model speaks in the release’s default voice; CreateSpeaker makes one from a recording. All three trained parts train here: the T2S and S2A models and the semantic tokenizer (Stage).

Constructors

Constructor Summary
WhisperSpeech(NeuralNetworkArchitecture<T> architecture) Calls the fully specified overload with the omitted parameters’ defaults.
WhisperSpeech(NeuralNetworkArchitecture<T> architecture, WhisperSpeechOptions options) Creates WhisperSpeech with its networks (randomly initialized).
WhisperSpeech(NeuralNetworkArchitecture<T> architecture, string modelPath, WhisperSpeechOptions options) Creates WhisperSpeech running an ONNX export.

Properties

Property Summary
Tensor<T> SpeakerEmbedding { get; set; } The speaker embedding the S2A model speaks with; null speaks in the release’s default voice.
WhisperSpeechConditions Conditions { get; set; } The conditioned S2A variant’s language, SNR and C50 (null: the language from the options, the rest missing).
bool TrainsOnSampleObjective { get; }
int CodebookSize { get; }
int CodecFrameRate { get; }
int MaxTextLength { get; } The longest text, in UTF-8 bytes (one position is the start token).
int NumCodebooks { get; }

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
T Train(string text, Tensor<T> recording) A step of Stage on a transcript and its recording at the model’s rate.
Tensor<T> CreateSpeaker(Tensor<T> recording, int sampleRate) A speaker embedding from a recording at any rate (extract_spk_emb: its first 30 seconds).
Tensor<T> DecodeFromTokens(Tensor<T> tokens)
Tensor<T> EncodeToTokens(Tensor<T> audio)
Tensor<T> Synthesize(string text)
int[] SemanticTokens(Tensor<T> recording, int sampleRate) The semantic tokens of a recording at any rate (the Whisper VQ tokenizer, at most 30 seconds).
int[] TextToSemanticTokens(string text) The semantic tokens the T2S model writes for text.
int[][] SemanticToAcousticTokens(IReadOnlyList<int> semantic) The EnCodec tokens [quantizers][frames] the S2A model writes for semantic tokens in SpeakerEmbedding.