MeloTTS<T>

Models & Types · AiDotNet.TextToSpeech.EndToEnd

MeloTTS (MyShell, myshell-ai/MeloTTS): a multilingual Bert-VITS2 — VITS with VITS2’s speaker-conditioned text encoder, transformer coupling flows, noise-scaled alignment search and duration discriminator, conditioned on per- symbol tones, languages and the features of a frozen per-language BERT, with a parity-tested front end for each of its languages.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → MeloTTS<T>

Implements: IEndToEndTts<T>

For Beginners

give it text and it speaks; train it with TtsTrainingSamples that carry the recording, its Text and the speaker. Released voices load with LoadCheckpoint plus LoadBertWeights for the language’s BERT.

How It Works

MeloTTS has no paper; its code is the reference (models.py, train.py, api.py, utils.get_text_for_tts_infer). Synthesis splits the text into sentences as split_utils does, reads each through the language’s front end and BERT (hidden_states[-3], each piece repeated for its phones), generates it with the stochastic and deterministic duration predictors mixed by SdpRatio, and joins the sentences with 50 ms of silence. Training takes MeloTTS’s step: the duration discriminator and the multi-period discriminator on a detached generation, then the generator on mel L1 × 45 + KL + duration + adversarial + feature matching (+ the duration adversarial term), each with its own AdamW.

Constructors

Constructor Summary
MeloTTS(NeuralNetworkArchitecture<T> architecture, MeloTTSOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a native, trainable MeloTTS.
MeloTTS(NeuralNetworkArchitecture<T> architecture, string modelPath, MeloTTSOptions options) Creates a native, trainable MeloTTS.

Properties

Property Summary
MeloLanguage Language { get; } The language this model speaks.
int HiddenDim { get; }
int MaxTextLength { get; }
int NumFlowSteps { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> Synthesize(string text)
void LoadBertWeights(string path) Loads the language’s BERT (its Hugging Face BertForMaskedLM or BertModel weights, e.g.
void LoadCheckpoint(string path) Loads a released MeloTTS checkpoint’s generator (checkpoint.pth’s model state dictionary, or that dictionary saved as safetensors).
void LoadDiscriminatorCheckpoint(string path) Loads a MeloTTS multi-period discriminator checkpoint (D_*.pth’s model state dictionary), which MeloTTS’s fine-tuning starts from alongside the generator.
void LoadDurationDiscriminatorCheckpoint(string path) Loads a MeloTTS duration discriminator checkpoint (DUR_*.pth’s model state dictionary).