SpeechT5<T>

Models & Types · AiDotNet.TextToSpeech.MultiModal

SpeechT5 (Ao et al. 2022): one Transformer encoder-decoder shared by speech and text, pre-trained jointly on unlabeled speech (HuBERT-style masked prediction plus reconstructing the log-mel spectrogram) and unlabeled text (BART-style infilling), with a cross-modal codebook aligning the two, then fine-tuned to read text aloud in the voice of a speaker x-vector.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → SpeechT5<T>

For Beginners

Give SpeechT5 a few seconds of someone speaking (or their x-vector) and text, and it reads the text in a voice like theirs.

How It Works

Text to speech runs the text encoder pre-net (a character embedding plus scaled positions), the shared encoder (post-norm layers with a relative position bias), and the speech decoder: its pre-net (Tacotron 2’s always-on dropout pre-net, scaled positions and the normalized x-vector), the shared decoder, r = 2 frames and a stop probability per step, and a five-layer convolutional post-net. HiFi-GAN (microsoft/speecht5_hifigan) turns the 16 kHz log-mel spectrogram into audio. The x-vector comes from speechbrain’s extractor (spkrec-xvect-voxceleb), as the release’s speaker embeddings do.

The paper is the default; the official code (microsoft/SpeechT5) fills what it leaves out, and the Hugging Face port’s stop rule is an option (StopCriterion).

Example

var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new SpeechT5<float>(architecture, new SpeechT5Options());
model.LoadReferenceWeights(new SpeechT5Checkpoints { Model = "speecht5_tts/pytorch_model.bin",
    Vocoder = "speecht5_hifigan/pytorch_model.bin", SpeakerEncoder = "spkrec-xvect-voxceleb/embedding_model.ckpt" });
model.Voice = model.CreateVoice(recording, 16000);
var audio = model.Synthesize("Hello world.");   // 16 kHz

Constructors

Constructor Summary
SpeechT5(NeuralNetworkArchitecture<T> architecture) Calls the fully specified overload with the omitted parameters’ defaults.
SpeechT5(NeuralNetworkArchitecture<T> architecture, SpeechT5Options options) Creates a native, trainable SpeechT5.
SpeechT5(NeuralNetworkArchitecture<T> architecture, string modelPath, SpeechT5Options options) Creates an ONNX-backed SpeechT5 for inference.

Properties

Property Summary
bool TrainsOnSampleObjective { get; }
int MaxTextLength { get; }
int Updates { get; } Updates taken so far in the current stage (the official num_updates, which the encoder freeze and the Gumbel temperature read).

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> GenerateSpectrogram(IReadOnlyList<int> tokens, TtsVoice<T> voice) The log-mel spectrogram [frames, mel] SpeechT5 generates for text token ids in a voice (before the vocoder).
Tensor<T> Synthesize(string text)
Tensor<T> Vocode(Tensor<T> mel) HiFi-GAN’s waveform [samples] of a log-mel spectrogram [frames, mel], normalized by the vocoder’s mean and scale first (normalize_before).
TtsVoice<T> CreateVoice(Tensor<T> recording, int sampleRate) A voice from a recording of the speaker at sampleRate: the recording resampled to 16 kHz and its speechbrain x-vector (the speaker embeddings the release was trained with).
int[] TokenizeText(string text) The token ids of text: its characters’ SentencePiece ids, then the end token.
void LoadReferenceWeights(SpeechT5Checkpoints checkpoints) Loads the release: the model (Hugging Face’s microsoft/speecht5_tts or an official checkpoint; the pre-training modules too when the checkpoint holds them), HiFi-GAN (microsoft/speecht5_hifigan) and the x-vector extractor (`spkrec-xvect…