MaskGCT<T>
Models & Types · AiDotNet.TextToSpeech.FlowDiffusion
MaskGCT (Wang et al. 2024): zero-shot text-to-speech by two masked generative transformers — text and a prompt’s semantic tokens to the target’s semantic tokens, then semantic tokens and the prompt’s acoustic tokens to the target’s acoustic tokens, layer by layer — with no text-speech alignment and no phone durations.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → MaskGCT<T>
Implements: ICodecTts<T>
For Beginners
Give MaskGCT a few seconds of someone speaking and what they said, and it speaks new text in that voice: it first sketches the speech’s content as coarse tokens, then fills in the sound in layers of finer tokens, refining many tokens at once in each pass.
How It Works
The paper fixes the design (§3.2): semantic tokens from w2v-BERT 2.0’s 17th hidden layer quantized by a VQ-VAE of
8,192 codes; acoustic tokens from a 24 kHz codec of twelve residual layers; a text-to-semantic transformer reading the
phones and the prompt’s semantic tokens as a prefix of the masked target, and a SoundStorm-like semantic-to-acoustic
transformer; both Llama-style, bidirectional, with adaptive RMSNorm on the step. The release (Amphion
models/tts/maskgct, amphion/MaskGCT) fills the rest: its G2P front ends, its decoding, the rule that sets the
target’s length from the prompt’s speaking rate, and the pair of semantic-to-acoustic models it ships — one for the
first acoustic layer and one for the other eleven given it. The paper’s flow-matching duration predictor (App. A.5)
is an option.
Synthesis needs a voice: a recording of the speaker and its transcript (CreateVoice). The text is
phonemized in Language — eSpeak NG (installed separately) for English, French, German and
Korean, Open JTalk with its dictionary for Japanese, MaskGCT’s Mandarin front end with the release’s polyphone model for
Chinese.
Example
var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new MaskGCT<float>(architecture, new MaskGCTOptions { Language = "en" });
model.LoadReferenceWeights("MaskGCT", "w2v-bert-2.0/model.safetensors", "wav2vec2bert_stats.pt");
model.Voice = model.CreateVoice(promptAudio, promptSampleRate, "The prompt's transcript.", "en");
var audio = model.Synthesize("Text to speak in that voice."); // 24 kHz
Constructors
| Constructor | Summary |
|---|---|
MaskGCT(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
MaskGCT(NeuralNetworkArchitecture<T> architecture, MaskGCTOptions options) |
Creates a native, trainable MaskGCT. |
MaskGCT(NeuralNetworkArchitecture<T> architecture, string modelPath, MaskGCTOptions options) |
Creates an ONNX-backed MaskGCT for inference. |
Properties
| Property | Summary |
|---|---|
bool TrainsOnSampleObjective { get; } |
|
int CodebookSize { get; } |
|
int CodecFrameRate { get; } |
Acoustic frames per second (50 at 24 kHz). |
int MaxTextLength { get; } |
|
int NumCodebooks { get; } |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> DecodeFromTokens(Tensor<T> tokens) |
|
Tensor<T> EncodeToTokens(Tensor<T> audio) |
|
Tensor<T> Synthesize(string text) |
|
TtsVoice<T> CreateVoice(Tensor<T> recording, int sampleRate, string transcript, string language) |
A voice for synthesis from a recording of the speaker at sampleRate and its transcript in language: the recording resampled to 24 kHz (its acoustic tokens are the prompt) and to 16 kHz (its semantic tokens are the prompt, its length set… |
ValueTuple<string, int[]> Phonemize(string text, string language) |
MaskGCT’s g2p_: the phone string and the phone ids of text in language. |
void LoadReferenceWeights(string directory, string semanticModelPath, string statisticsPath) |
Loads the release: an amphion/MaskGCT directory (semantic_codec/model.safetensors, acoustic_codec/model.safetensors and model_1.safetensors, t2s_model/model.safetensors, s2a_model/s2a_model_1layer/model.safetensors, `s2a_model/s… |
void LoadSemanticModel(string path) |
Loads w2v-BERT 2.0 (facebook/w2v-bert-2.0’s model.safetensors); the layers past SemanticLayer are never run and not read. |
void LoadStatistics(string path) |
Loads the semantic feature statistics (wav2vec2bert_stats.pt: per-channel mean and var). |