MegaTTS<T>
Models & Types · AiDotNet.TextToSpeech.Latest
Mega-TTS (Jiang et al. 2023): zero-shot speech from a disentangled mel-spectrogram model. A content encoder reads the phonemes, a prosody encoder turns the low band of a mel spectrogram into one VQ code per phoneme, a timbre encoder averages a reference recording into one vector, and a GAN-trained decoder makes the mel spectrogram from the three; a prosody language model (P-LLM) continues the prompt’s prosody codes for the new text, and HiFi-GAN V1 makes the audio.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → MegaTTS<T>
Implements: IEndToEndTts<T>
For Beginners
give Mega-TTS a few seconds of someone speaking (Voice’s
ReferenceAudio) with that recording’s phonemes and their durations
(ReferenceTokens, ReferenceDurations), and it says new text in
that voice and with that person’s way of speaking. Training goes in two stages: the acoustic model on recordings with
phoneme durations, then the prosody language model (Stage).
How It Works
The paper’s model with its Table 7 sizes. Mega-TTS has no released code; what the paper leaves unstated comes from the authors’ NATSpeech code base, which it builds on: the text front end, the mel features, the encoder layers, the duration predictor and its loss, and SyntaSpeech’s multi-window discriminator and its training.
Example
var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new MegaTTS<float>(architecture, new MegaTTSOptions());
model.LoadVocoderWeights("generator_v1"); // a HiFi-GAN V1 generator trained on this model's mel spectrograms
model.Voice = new TtsVoice<float>
{
ReferenceAudio = prompt22kHz,
ReferenceTokens = model.TextToTokens(promptTranscript),
ReferenceDurations = promptDurations, // frames per token, from a forced aligner
};
var audio = model.Synthesize("Hello, how are you doing today?");
Constructors
| Constructor | Summary |
|---|---|
MegaTTS(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
MegaTTS(NeuralNetworkArchitecture<T> architecture, MegaTTSOptions options) |
Creates a native, trainable Mega-TTS. |
MegaTTS(NeuralNetworkArchitecture<T> architecture, string modelPath, MegaTTSOptions options) |
Creates an ONNX-backed Mega-TTS for inference. |
Properties
| Property | Summary |
|---|---|
bool TrainsOnSampleObjective { get; } |
|
int HiddenDim { get; } |
|
int MaxTextLength { get; } |
|
int NumFlowSteps { get; } |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
T TrainProsodyModel(IReadOnlyList<TtsTrainingSample<T>> sentences) |
Trains the P-LLM on sentences of one speaker read as one sequence (the paper’s Table 7 trains on 7 contextual sentences): each sentence’s prosody codes from the frozen prosody encoder and its content from the frozen content encoder, joined;… |
Tensor<T> Synthesize(string text) |
|
void LoadVocoderWeights(string path) |
Loads a HiFi-GAN V1 generator checkpoint in the reference’s format (jik876/hifi-gan’s {"generator": state_dict}, or the bare state dict), trained on this model’s mel spectrograms. |