MegaTTS3<T>
Models & Types · AiDotNet.TextToSpeech.Latest
MegaTTS 3 (Jiang et al. 2025): zero-shot speech from a sparse-alignment latent diffusion transformer. A prompt recording is aligned by a Whisper aligner into phones, tones and durations and encoded by the WaveVAE; the text is normalized, chunked and turned into phones by a Qwen2 grapheme-to-phoneme model; an autoregressive duration model continues the prompt’s durations; the diffusion transformer generates the WaveVAE latent after the prompt with multi-condition classifier-free guidance; the WaveVAE decodes 24 kHz audio, matched to the prompt’s loudness.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → MegaTTS3<T>
Implements: IEndToEndTts<T>
For Beginners
give MegaTTS 3 a few seconds of someone speaking (Voice’s
reference recording) and a sentence, and it says the sentence in that voice.
How It Works
Inference is the release’s (bytedance/MegaTTS3, tts/infer_cli.py and tts/frontend_function.py), with
WeTextProcessing’s normalizers, langdetect, librosa’s soxr resampling and pyloudnorm ported; training is the paper’s,
which the release does not include: rectified flow on the latent’s masked target region, sparse alignment anchors, and
the prompt and text dropped for classifier-free guidance.
Example
var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new MegaTTS3<float>(architecture, new MegaTTS3Options());
model.LoadReferenceWeights("checkpoints"); // ByteDance/MegaTTS3: diffusion_transformer, duration_lm, aligner_lm, g2p, wavvae
model.PromptLatent = MegaTTS3<float>.LoadLatent("prompt.npy"); // the release withholds the WaveVAE encoder
model.Voice = new TtsVoice<float> { ReferenceAudio = prompt24kHz };
var audio = model.Synthesize("Hello, how are you doing today?");
Constructors
| Constructor | Summary |
|---|---|
MegaTTS3(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
MegaTTS3(NeuralNetworkArchitecture<T> architecture, MegaTTS3Options options) |
Creates a native, trainable MegaTTS 3. |
MegaTTS3(NeuralNetworkArchitecture<T> architecture, string modelPath, MegaTTS3Options options) |
Creates an ONNX-backed MegaTTS 3 for inference. |
Properties
| Property | Summary |
|---|---|
Tensor<T> PromptLatent { get; set; } |
The prompt’s WaveVAE latent [frames, channels] to use instead of encoding the voice’s recording (the release’s .npy latents, for its decoder-only checkpoint); null encodes the recording. |
bool TrainsOnSampleObjective { get; } |
|
int MaxTextLength { get; } |
|
int NumFlowSteps { get; } |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> LoadLatent(string path) |
Reads a prompt latent saved by the release (a .npy of [1, frames, channels] or [frames, channels]). |
Tensor<T> Synthesize(string text) |
|
void LoadReferenceWeights(string root) |
Loads the release’s checkpoint folder (ByteDance/MegaTTS3): diffusion_transformer/model_only_last.ckpt, duration_lm/model_only_last.ckpt, aligner_lm/model_only_last.ckpt, g2p/ (Qwen2’s model.safetensors and tokenizer.json) and `… |