SPEARTTS<T>
Models & Types · AiDotNet.TextToSpeech.CodecBased
SPEAR-TTS: multi-speaker text-to-speech trained with minimal supervision (Kharitonov et al. 2023), in two stages: “reading” text into semantic tokens and “speaking” them as acoustic tokens.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → SPEARTTS<T>
Implements: ICodecTts<T>
For Beginners
SPEAR-TTS first learns how speech is built from hours of recordings with no transcripts, then needs only a few minutes of transcribed speech to learn to read, and speaks in any voice it hears for three seconds.
How It Works
Reading, S1 (§4): a T5-Large encoder–decoder from phonemes to w2v-BERT semantic tokens (512 clusters, 25 Hz,
consecutive repetitions removed). It is pretrained on audio alone to reconstruct each semantic sequence from a copy
with every token deleted with probability 0.6 (Pretraining), then finetuned on a
little parallel data (Reading): only the encoder’s embeddings and its lower
layers learn, the upper encoder and the whole decoder stay as pretrained. Backtranslation (§4.2) finetunes a copy of
the pretrained model the other way, semantic tokens to text (Backtranslation);
Backtranscribe then turns untranscribed audio into synthetic parallel data for the reading stage.
Synthesis reads with beam search (beam 10).
Speaking, S2 (§5): a decoder-only Transformer from semantic to acoustic tokens, the first 3 levels of a
SoundStream codec interleaved per frame. Training takes two non-overlapping windows of each utterance: the prompt’s
semantic tokens, the target’s semantic tokens and the prompt’s acoustic tokens are the prefix, and the model learns
the target’s acoustic tokens, so the voice and recording conditions of a 3-second prompt carry over to what it
speaks. Synthesis samples at temperature 0.75 with Voice as the prompt.
No official code exists. Gaps in S1’s procedure follow lucidrains/spear-tts-pytorch and S2’s sequence layout
follows AudioLM’s coarse stage (AudioLmCore); each choice is listed in
docs/text-to-speech/paper-fidelity-decisions.md.
Example
var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional,
NeuralNetworkTaskType.Generative, inputSize: 1, outputSize: 1);
var model = new SPEARTTS<float>(architecture, new SPEARTTSOptions(), semanticTokenizer: w2vBert);
model.Voice = model.CreateVoice(threeSecondRecording);
Tensor<float> audio = model.Synthesize("Hello there.");
Constructors
| Constructor | Summary |
|---|---|
SPEARTTS(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
SPEARTTS(NeuralNetworkArchitecture<T> architecture, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates an ONNX-backed SPEAR-TTS for inference. |
SPEARTTS(NeuralNetworkArchitecture<T> architecture, SPEARTTSOptions options) |
Creates an ONNX-backed SPEAR-TTS for inference. |
SPEARTTS(NeuralNetworkArchitecture<T> architecture, SPEARTTSOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates an ONNX-backed SPEAR-TTS with explicit options. |
SPEARTTS(NeuralNetworkArchitecture<T> architecture, SPEARTTSOptions options, W2vBert<T> semanticTokenizer) |
Creates an ONNX-backed SPEAR-TTS with explicit options. |
SPEARTTS(NeuralNetworkArchitecture<T> architecture, SPEARTTSOptions options, W2vBert<T> semanticTokenizer, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a native, trainable SPEAR-TTS. |
SPEARTTS(NeuralNetworkArchitecture<T> architecture, W2vBert<T> semanticTokenizer) |
Creates an ONNX-backed SPEAR-TTS for inference. |
SPEARTTS(NeuralNetworkArchitecture<T> architecture, W2vBert<T> semanticTokenizer, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates an ONNX-backed SPEAR-TTS with explicit options. |
SPEARTTS(NeuralNetworkArchitecture<T> architecture, string modelPath) |
Creates an ONNX-backed SPEAR-TTS for inference. |
SPEARTTS(NeuralNetworkArchitecture<T> architecture, string modelPath, SPEARTTSOptions options) |
Creates an ONNX-backed SPEAR-TTS with explicit options. |
Properties
| Property | Summary |
|---|---|
SPEARTTSTrainingStage CurrentStage { get; set; } |
The network and task a training step runs; Reading, training on parallel data, until set. |
SoundStream<T> Codec { get; } |
The SoundStream codec whose codes SPEAR-TTS speaks. |
int CodebookSize { get; } |
|
int CodecFrameRate { get; } |
|
int CodecTokenCodebooks { get; } |
|
int CodecTokenVocabulary { get; } |
|
int MaxTextLength { get; } |
|
int NumCodebooks { get; } |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> DecodeFromTokens(Tensor<T> tokens) |
The waveform of codec codes [frames, Q]. |
Tensor<T> EncodeToTokens(Tensor<T> audio) |
The codec’s first NumCodebooks levels of a recording, [frames, Q]. |
TtsVoice<T> CreateVoice(Tensor<T> recording, int[] semanticTokens) |
A voice for Voice: a recording of the speaker, of which the first PromptSeconds are the prompt, with its semantic tokens at 25 Hz (from the semantic tokenizer when semanticTokens is null). |
int[] EncodePhonemes(IEnumerable<string> symbols) |
S1’s token id of each phoneme symbol (espeak-ng’s, as EnglishG2P emits them); symbols outside the table are dropped. |
int[] ReadText(Tensor<T> textIds) |
S1’s semantic tokens (deduplicated, 25 Hz) for text ids, by beam search (§7.4). |
string[] Backtranscribe(IReadOnlyList<int> semanticTokens) |
Backtranslation (§4.2): the phonemes the backtranslation model reads out of an utterance’s semantic tokens (25 Hz; repetitions are removed here), by the same beam search as reading. |
string[] DecodePhonemes(IEnumerable<int> ids) |
The phoneme symbols of S1 token ids (special and semantic ids are skipped). |
void StartBacktranslationFromPretrained() |
Copies the pretrained S1 into the backtranslation model (§4.2: “the same pretrained checkpoint P is finetuned for different purposes”). |