WhisperSpeech<T>
Models & Types · AiDotNet.TextToSpeech.MultiModal
WhisperSpeech (Collabora 2024): text-to-speech by inverting Whisper, after SPEAR-TTS (Kharitonov et al. 2023). A text-to-semantic transformer writes the semantic tokens Whisper’s quantized encoder would hear for the text, a semantic-to-acoustic transformer writes EnCodec’s tokens for them in a speaker’s voice (an ECAPA-TDNN embedding), and Vocos turns the EnCodec tokens into 24 kHz speech.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → WhisperSpeech<T>
Implements: ICodecTts<T>
For Beginners
WhisperSpeech first decides “what is said” as the tokens a speech recognizer would hear, then “how it sounds” as audio-codec tokens in a chosen voice, and finally turns those into a waveform.
How It Works
Built as the release (collabora/WhisperSpeech) runs it; see WhisperSpeechOptions and
docs/text-to-speech/paper-fidelity-decisions.md. Text goes in as UTF-8 bytes; speech comes out at 24 kHz. Without
a SpeakerEmbedding the model speaks in the release’s default voice; CreateSpeaker makes one
from a recording. All three trained parts train here: the T2S and S2A models and the semantic tokenizer
(Stage).
Constructors
| Constructor | Summary |
|---|---|
WhisperSpeech(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
WhisperSpeech(NeuralNetworkArchitecture<T> architecture, WhisperSpeechOptions options) |
Creates WhisperSpeech with its networks (randomly initialized). |
WhisperSpeech(NeuralNetworkArchitecture<T> architecture, string modelPath, WhisperSpeechOptions options) |
Creates WhisperSpeech running an ONNX export. |
Properties
| Property | Summary |
|---|---|
Tensor<T> SpeakerEmbedding { get; set; } |
The speaker embedding the S2A model speaks with; null speaks in the release’s default voice. |
WhisperSpeechConditions Conditions { get; set; } |
The conditioned S2A variant’s language, SNR and C50 (null: the language from the options, the rest missing). |
bool TrainsOnSampleObjective { get; } |
|
int CodebookSize { get; } |
|
int CodecFrameRate { get; } |
|
int MaxTextLength { get; } |
The longest text, in UTF-8 bytes (one position is the start token). |
int NumCodebooks { get; } |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
T Train(string text, Tensor<T> recording) |
A step of Stage on a transcript and its recording at the model’s rate. |
Tensor<T> CreateSpeaker(Tensor<T> recording, int sampleRate) |
A speaker embedding from a recording at any rate (extract_spk_emb: its first 30 seconds). |
Tensor<T> DecodeFromTokens(Tensor<T> tokens) |
|
Tensor<T> EncodeToTokens(Tensor<T> audio) |
|
Tensor<T> Synthesize(string text) |
|
int[] SemanticTokens(Tensor<T> recording, int sampleRate) |
The semantic tokens of a recording at any rate (the Whisper VQ tokenizer, at most 30 seconds). |
int[] TextToSemanticTokens(string text) |
The semantic tokens the T2S model writes for text. |
int[][] SemanticToAcousticTokens(IReadOnlyList<int> semantic) |
The EnCodec tokens [quantizers][frames] the S2A model writes for semantic tokens in SpeakerEmbedding. |