MatchaTTS<T>

Models & Types · AiDotNet.TextToSpeech.FlowDiffusion

Matcha-TTS: a non-autoregressive acoustic model whose decoder is trained with optimal-transport conditional flow matching and samples a mel spectrogram in a few ODE steps.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → MatchaTTS<T>

Implements: IAcousticModel<T>, IEndToEndTts<T>

For Beginners

Matcha-TTS learns a straight-line path from random noise to a spectrogram, so turning noise into speech takes only a handful of steps instead of the hundreds diffusion models need.

How It Works

References: “Matcha-TTS: A Fast TTS Architecture with Conditional Flow Matching” (Mehta et al., ICASSP 2024) and its reference implementation (shivammehta25/Matcha-TTS) for what the paper leaves unstated.

The encoder is Grad-TTS’s (a Glow-TTS text encoder) with rotary position embeddings in place of relative ones: an embedding scaled by √d, a 3-layer convolutional pre-net, 6 post-norm Transformer layers, a projection to per-token mel means μ and a duration predictor on the stop-gradient encoder output. Training aligns μ to the mel by monotonic alignment search. The decoder MatchaDecoder is a 1-D U-Net with Transformer blocks that estimates the vector field v_θ(x_t, μ, t) of the OT-CFM path x_t = (1 − (1 − σ_min) t) x₀ + t x₁ with target u = x₁ − (1 − σ_min) x₀, x₀ ~ N(0, I). The loss is the duration loss + the prior loss + the CFM loss on mel spectrograms normalized by the dataset mean and standard deviation; Adam at 1e-4, gradients clipped to norm 5. Synthesis draws x₀ ~ N(0, τ² I) and integrates the field from t = 0 to 1 with Euler steps, then denormalizes.

The model aligns itself, so a plain Train(tokens, mel) is its full training signal. It outputs a mel spectrogram; the paper pairs it with a HiFi-GAN vocoder.

Example

var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.Regression,
    inputSize: 200, outputSize: 80);
            
// ONNX inference mode with pre-trained model
var model = new MatchaTTS<double>(architecture, "matcha_tts.onnx");
            
// Training mode with native layers
var trainModel = new MatchaTTS<double>(architecture, new MatchaTTSOptions());
var mel = trainModel.TextToMel("hello world");

Constructors

Constructor Summary
MatchaTTS(NeuralNetworkArchitecture<T> architecture, MatchaTTSOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
MatchaTTS(NeuralNetworkArchitecture<T> architecture, string modelPath, MatchaTTSOptions options)

Properties

Property Summary
int FftSize { get; }
int HiddenDim { get; }
int HopSize { get; }
int MaxTextLength { get; }
int MelChannels { get; }
int NumFlowSteps { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> TextToMel(string text) Generates a mel spectrogram from text (the paper pairs it with HiFi-GAN).