MatchaTTS<T>
Models & Types · AiDotNet.TextToSpeech.FlowDiffusion
Matcha-TTS: a non-autoregressive acoustic model whose decoder is trained with optimal-transport conditional flow matching and samples a mel spectrogram in a few ODE steps.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → MatchaTTS<T>
Implements: IAcousticModel<T>, IEndToEndTts<T>
For Beginners
Matcha-TTS learns a straight-line path from random noise to a spectrogram, so turning noise into speech takes only a handful of steps instead of the hundreds diffusion models need.
How It Works
References: “Matcha-TTS: A Fast TTS Architecture with Conditional Flow Matching” (Mehta et al., ICASSP 2024) and its reference implementation (shivammehta25/Matcha-TTS) for what the paper leaves unstated.
The encoder is Grad-TTS’s (a Glow-TTS text encoder) with rotary position embeddings in place of relative ones: an
embedding scaled by √d, a 3-layer convolutional pre-net, 6 post-norm Transformer layers, a projection to per-token
mel means μ and a duration predictor on the stop-gradient encoder output. Training aligns μ to the mel by monotonic
alignment search. The decoder MatchaDecoder is a 1-D U-Net with Transformer blocks that estimates the vector
field v_θ(x_t, μ, t) of the OT-CFM path x_t = (1 − (1 − σ_min) t) x₀ + t x₁ with target
u = x₁ − (1 − σ_min) x₀, x₀ ~ N(0, I). The loss is the duration loss + the prior loss + the CFM loss on
mel spectrograms normalized by the dataset mean and standard deviation; Adam at 1e-4, gradients clipped to norm 5.
Synthesis draws x₀ ~ N(0, τ² I) and integrates the field from t = 0 to 1 with Euler steps, then denormalizes.
The model aligns itself, so a plain Train(tokens, mel) is its full training signal. It outputs a mel
spectrogram; the paper pairs it with a HiFi-GAN vocoder.
Example
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.Regression,
inputSize: 200, outputSize: 80);
// ONNX inference mode with pre-trained model
var model = new MatchaTTS<double>(architecture, "matcha_tts.onnx");
// Training mode with native layers
var trainModel = new MatchaTTS<double>(architecture, new MatchaTTSOptions());
var mel = trainModel.TextToMel("hello world");
Constructors
| Constructor | Summary |
|---|---|
MatchaTTS(NeuralNetworkArchitecture<T> architecture, MatchaTTSOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
MatchaTTS(NeuralNetworkArchitecture<T> architecture, string modelPath, MatchaTTSOptions options) |
Properties
| Property | Summary |
|---|---|
int FftSize { get; } |
|
int HiddenDim { get; } |
|
int HopSize { get; } |
|
int MaxTextLength { get; } |
|
int MelChannels { get; } |
|
int NumFlowSteps { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> TextToMel(string text) |
Generates a mel spectrogram from text (the paper pairs it with HiFi-GAN). |