MeloTTS<T>
Models & Types · AiDotNet.TextToSpeech.EndToEnd
MeloTTS (MyShell, myshell-ai/MeloTTS): a multilingual Bert-VITS2 — VITS with VITS2’s speaker-conditioned text encoder, transformer coupling flows, noise-scaled alignment search and duration discriminator, conditioned on per- symbol tones, languages and the features of a frozen per-language BERT, with a parity-tested front end for each of its languages.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → MeloTTS<T>
Implements: IEndToEndTts<T>
For Beginners
give it text and it speaks; train it with TtsTrainingSamples that carry
the recording, its Text and the speaker. Released voices load with
LoadCheckpoint plus LoadBertWeights for the language’s BERT.
How It Works
MeloTTS has no paper; its code is the reference (models.py, train.py, api.py,
utils.get_text_for_tts_infer). Synthesis splits the text into sentences as split_utils does, reads
each through the language’s front end and BERT (hidden_states[-3], each piece repeated for its phones),
generates it with the stochastic and deterministic duration predictors mixed by SdpRatio,
and joins the sentences with 50 ms of silence. Training takes MeloTTS’s step: the duration discriminator and the
multi-period discriminator on a detached generation, then the generator on mel L1 × 45 + KL + duration +
adversarial + feature matching (+ the duration adversarial term), each with its own AdamW.
Constructors
| Constructor | Summary |
|---|---|
MeloTTS(NeuralNetworkArchitecture<T> architecture, MeloTTSOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a native, trainable MeloTTS. |
MeloTTS(NeuralNetworkArchitecture<T> architecture, string modelPath, MeloTTSOptions options) |
Creates a native, trainable MeloTTS. |
Properties
| Property | Summary |
|---|---|
MeloLanguage Language { get; } |
The language this model speaks. |
int HiddenDim { get; } |
|
int MaxTextLength { get; } |
|
int NumFlowSteps { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> Synthesize(string text) |
|
void LoadBertWeights(string path) |
Loads the language’s BERT (its Hugging Face BertForMaskedLM or BertModel weights, e.g. |
void LoadCheckpoint(string path) |
Loads a released MeloTTS checkpoint’s generator (checkpoint.pth’s model state dictionary, or that dictionary saved as safetensors). |
void LoadDiscriminatorCheckpoint(string path) |
Loads a MeloTTS multi-period discriminator checkpoint (D_*.pth’s model state dictionary), which MeloTTS’s fine-tuning starts from alongside the generator. |
void LoadDurationDiscriminatorCheckpoint(string path) |
Loads a MeloTTS duration discriminator checkpoint (DUR_*.pth’s model state dictionary). |