MegaTTS2<T>

Models & Types · AiDotNet.TextToSpeech.Latest

Mega-TTS 2 (Jiang et al. 2024): zero-shot speech from prompts of any length. An acoustic autoencoder splits speech into content (a Transformer over the phonemes), prosody (a VQ encoder that pools the mel spectrogram eight times into codes) and timbre (the multi-reference timbre encoder, whose content attends to the joined prompt recordings); a prosody latent language model (P-LLM) continues the prompts’ prosody codes for the new text, an autoregressive duration model their phoneme durations, and HiFi-GAN V1 makes the audio. Prosody interpolation mixes the P-LLM’s distributions for two speakers’ prosody prompts.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → MegaTTS2<T>

Implements: IEndToEndTts<T>

For Beginners

give Mega-TTS 2 one or more recordings of someone speaking (Voice, or PromptSentences for several), each with its phonemes and their durations, and it says new text in that voice and with that person’s way of speaking; the more sentences, the closer it gets. Set ProsodyPrompts to recordings of another speaker to borrow their speaking style while keeping the voice. Training goes in two stages: the acoustic model, then the language models (Stage).

How It Works

The paper’s model with its Table 5 sizes. Mega-TTS 2 has no released code; what the paper leaves unstated comes from the authors’ code: NATSpeech’s encoder layers and convolution stacks, SyntaSpeech’s multi-length discriminator, and the decoder-only Transformer of MegaTTS 3’s released duration predictor (whose code embedding of K + 2 entries is the paper’s 1026), and from CVQ-VAE’s code for its quantizer.

Example

var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new MegaTTS2<float>(architecture, new MegaTTS2Options());
model.LoadVocoderWeights("generator_v1");   // a HiFi-GAN V1 generator trained on this model's mel spectrograms
model.PromptSentences = new[]
{
    new TtsVoice<float> { ReferenceAudio = clip1, ReferenceTokens = model.TextToTokens(text1), ReferenceDurations = durations1 },
    new TtsVoice<float> { ReferenceAudio = clip2, ReferenceTokens = model.TextToTokens(text2), ReferenceDurations = durations2 },
};
var audio = model.Synthesize("Hello, how are you doing today?");

Constructors

Constructor Summary
MegaTTS2(NeuralNetworkArchitecture<T> architecture) Calls the fully specified overload with the omitted parameters’ defaults.
MegaTTS2(NeuralNetworkArchitecture<T> architecture, MegaTTS2Options options) Creates a native, trainable Mega-TTS 2.
MegaTTS2(NeuralNetworkArchitecture<T> architecture, string modelPath, MegaTTS2Options options) Creates an ONNX-backed Mega-TTS 2 for inference.

Properties

Property Summary
IReadOnlyList<TtsVoice<T>> PromptSentences { get; set; } The target speaker’s prompt sentences, joined for the timbre encoder and read in order by the language models; null takes Voice alone.
IReadOnlyList<TtsVoice<T>> ProsodyPrompts { get; set; } Another speaker’s prompt sentences, whose prosody the P-LLM mixes in with weight ProsodyInterpolation (§3.4); each needs its phonemes and durations.
bool TrainsOnSampleObjective { get; }
int HiddenDim { get; }
int MaxTextLength { get; }
int NumFlowSteps { get; }

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
T TrainLanguageModels(IReadOnlyList<IReadOnlyList<TtsTrainingSample<T>>> speakers) One step of the second stage on sentences grouped by speaker (§3.3): each sentence’s prosody codes and content from the frozen autoencoder, every speaker’s sentences one sequence with its own positions, attention kept within each speaker; t…
Tensor<T> Synthesize(string text)
void LoadVocoderWeights(string path) Loads a HiFi-GAN V1 generator checkpoint in the reference’s format (jik876/hifi-gan’s {"generator": state_dict}, or the bare state dict), trained on this model’s mel spectrograms.