FlowMatchingTtsModelBase<T>

Base Classes · AiDotNet.TextToSpeech.FlowDiffusion

The text-guided speech-infilling conditional flow matching shared by E2 TTS and F5-TTS (Chen et al. 2024, §2–3; reference SWivid/F5-TTS model/cfm.py): characters padded with filler tokens to the mel length, a masked copy of the mel as the audio condition, a backbone predicting the flow x₁ − x₀, classifier-free guidance and sway sampling.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → FlowMatchingTtsModelBase<T>

Known derived types: E2TTS<T>, F5TTS<T>

How It Works

Training draws x₀ ~ N(0, I) and t ~ U[0, 1], forms ψ_t = (1 − t) x₀ + t x₁, zeroes a random contiguous span of 70–100 % of the frames in the audio condition, drops the audio condition with probability 0.3 and both audio and text with probability 0.2 (the unconditional model for CFG), and minimizes the mean squared error to the flow over the masked span. Synthesis integrates the predicted field with Euler steps over sway-sampled flow steps t = u + s (cos(π u / 2) − 1 + u) (Eq. 7), with CFG v + α (v − v_uncond) (Eq. 6), keeping the prompt frames. With Voice carrying a reference mel and its transcript tokens the reference prefixes both the condition and the text and sets the speaking rate; otherwise the condition is silent and the length is the character count times FramesPerCharacter.

Methods

Method Summary
LayerInputDomain GetInputDomain(int[] inputShape)
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
void Train(Tensor<T> input, Tensor<T> expectedOutput)