FlowMatchingTtsModelBase<T>
Base Classes · AiDotNet.TextToSpeech.FlowDiffusion
The text-guided speech-infilling conditional flow matching shared by E2 TTS and F5-TTS (Chen et al. 2024, §2–3;
reference SWivid/F5-TTS model/cfm.py): characters padded with filler tokens to the mel length, a masked copy of
the mel as the audio condition, a backbone predicting the flow x₁ − x₀, classifier-free guidance and sway
sampling.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → FlowMatchingTtsModelBase<T>
Known derived types: E2TTS<T>, F5TTS<T>
How It Works
Training draws x₀ ~ N(0, I) and t ~ U[0, 1], forms ψ_t = (1 − t) x₀ + t x₁, zeroes a random contiguous span of
70–100 % of the frames in the audio condition, drops the audio condition with probability 0.3 and both audio and text
with probability 0.2 (the unconditional model for CFG), and minimizes the mean squared error to the flow over the
masked span. Synthesis integrates the predicted field with Euler steps over sway-sampled flow steps
t = u + s (cos(π u / 2) − 1 + u) (Eq. 7), with CFG v + α (v − v_uncond) (Eq. 6), keeping the prompt
frames. With Voice carrying a reference mel and its transcript tokens the reference
prefixes both the condition and the text and sets the speaking rate; otherwise the condition is silent and the length
is the character count times FramesPerCharacter.
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
void Train(Tensor<T> input, Tensor<T> expectedOutput) |