SpeechT5<T>
Models & Types · AiDotNet.TextToSpeech.MultiModal
SpeechT5 (Ao et al. 2022): one Transformer encoder-decoder shared by speech and text, pre-trained jointly on unlabeled speech (HuBERT-style masked prediction plus reconstructing the log-mel spectrogram) and unlabeled text (BART-style infilling), with a cross-modal codebook aligning the two, then fine-tuned to read text aloud in the voice of a speaker x-vector.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → SpeechT5<T>
For Beginners
Give SpeechT5 a few seconds of someone speaking (or their x-vector) and text, and it reads the text in a voice like theirs.
How It Works
Text to speech runs the text encoder pre-net (a character embedding plus scaled positions), the shared encoder
(post-norm layers with a relative position bias), and the speech decoder: its pre-net (Tacotron 2’s always-on dropout
pre-net, scaled positions and the normalized x-vector), the shared decoder, r = 2 frames and a stop probability per
step, and a five-layer convolutional post-net. HiFi-GAN (microsoft/speecht5_hifigan) turns the 16 kHz log-mel
spectrogram into audio. The x-vector comes from speechbrain’s extractor (spkrec-xvect-voxceleb), as the
release’s speaker embeddings do.
The paper is the default; the official code (microsoft/SpeechT5) fills what it leaves out, and the Hugging
Face port’s stop rule is an option (StopCriterion).
Example
var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new SpeechT5<float>(architecture, new SpeechT5Options());
model.LoadReferenceWeights(new SpeechT5Checkpoints { Model = "speecht5_tts/pytorch_model.bin",
Vocoder = "speecht5_hifigan/pytorch_model.bin", SpeakerEncoder = "spkrec-xvect-voxceleb/embedding_model.ckpt" });
model.Voice = model.CreateVoice(recording, 16000);
var audio = model.Synthesize("Hello world."); // 16 kHz
Constructors
| Constructor | Summary |
|---|---|
SpeechT5(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
SpeechT5(NeuralNetworkArchitecture<T> architecture, SpeechT5Options options) |
Creates a native, trainable SpeechT5. |
SpeechT5(NeuralNetworkArchitecture<T> architecture, string modelPath, SpeechT5Options options) |
Creates an ONNX-backed SpeechT5 for inference. |
Properties
| Property | Summary |
|---|---|
bool TrainsOnSampleObjective { get; } |
|
int MaxTextLength { get; } |
|
int Updates { get; } |
Updates taken so far in the current stage (the official num_updates, which the encoder freeze and the Gumbel temperature read). |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> GenerateSpectrogram(IReadOnlyList<int> tokens, TtsVoice<T> voice) |
The log-mel spectrogram [frames, mel] SpeechT5 generates for text token ids in a voice (before the vocoder). |
Tensor<T> Synthesize(string text) |
|
Tensor<T> Vocode(Tensor<T> mel) |
HiFi-GAN’s waveform [samples] of a log-mel spectrogram [frames, mel], normalized by the vocoder’s mean and scale first (normalize_before). |
TtsVoice<T> CreateVoice(Tensor<T> recording, int sampleRate) |
A voice from a recording of the speaker at sampleRate: the recording resampled to 16 kHz and its speechbrain x-vector (the speaker embeddings the release was trained with). |
int[] TokenizeText(string text) |
The token ids of text: its characters’ SentencePiece ids, then the end token. |
void LoadReferenceWeights(SpeechT5Checkpoints checkpoints) |
Loads the release: the model (Hugging Face’s microsoft/speecht5_tts or an official checkpoint; the pre-training modules too when the checkpoint holds them), HiFi-GAN (microsoft/speecht5_hifigan) and the x-vector extractor (`spkrec-xvect… |