MegaTTS3<T>

Models & Types · AiDotNet.TextToSpeech.Latest

MegaTTS 3 (Jiang et al. 2025): zero-shot speech from a sparse-alignment latent diffusion transformer. A prompt recording is aligned by a Whisper aligner into phones, tones and durations and encoded by the WaveVAE; the text is normalized, chunked and turned into phones by a Qwen2 grapheme-to-phoneme model; an autoregressive duration model continues the prompt’s durations; the diffusion transformer generates the WaveVAE latent after the prompt with multi-condition classifier-free guidance; the WaveVAE decodes 24 kHz audio, matched to the prompt’s loudness.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → MegaTTS3<T>

Implements: IEndToEndTts<T>

For Beginners

give MegaTTS 3 a few seconds of someone speaking (Voice’s reference recording) and a sentence, and it says the sentence in that voice.

How It Works

Inference is the release’s (bytedance/MegaTTS3, tts/infer_cli.py and tts/frontend_function.py), with WeTextProcessing’s normalizers, langdetect, librosa’s soxr resampling and pyloudnorm ported; training is the paper’s, which the release does not include: rectified flow on the latent’s masked target region, sparse alignment anchors, and the prompt and text dropped for classifier-free guidance.

Example

var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new MegaTTS3<float>(architecture, new MegaTTS3Options());
model.LoadReferenceWeights("checkpoints");   // ByteDance/MegaTTS3: diffusion_transformer, duration_lm, aligner_lm, g2p, wavvae
model.PromptLatent = MegaTTS3<float>.LoadLatent("prompt.npy");   // the release withholds the WaveVAE encoder
model.Voice = new TtsVoice<float> { ReferenceAudio = prompt24kHz };
var audio = model.Synthesize("Hello, how are you doing today?");

Constructors

Constructor Summary
MegaTTS3(NeuralNetworkArchitecture<T> architecture) Calls the fully specified overload with the omitted parameters’ defaults.
MegaTTS3(NeuralNetworkArchitecture<T> architecture, MegaTTS3Options options) Creates a native, trainable MegaTTS 3.
MegaTTS3(NeuralNetworkArchitecture<T> architecture, string modelPath, MegaTTS3Options options) Creates an ONNX-backed MegaTTS 3 for inference.

Properties

Property Summary
Tensor<T> PromptLatent { get; set; } The prompt’s WaveVAE latent [frames, channels] to use instead of encoding the voice’s recording (the release’s .npy latents, for its decoder-only checkpoint); null encodes the recording.
bool TrainsOnSampleObjective { get; }
int MaxTextLength { get; }
int NumFlowSteps { get; }

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> LoadLatent(string path) Reads a prompt latent saved by the release (a .npy of [1, frames, channels] or [frames, channels]).
Tensor<T> Synthesize(string text)
void LoadReferenceWeights(string root) Loads the release’s checkpoint folder (ByteDance/MegaTTS3): diffusion_transformer/model_only_last.ckpt, duration_lm/model_only_last.ckpt, aligner_lm/model_only_last.ckpt, g2p/ (Qwen2’s model.safetensors and tokenizer.json) and `…