F5TTS<T>

Models & Types · AiDotNet.TextToSpeech.FlowDiffusion

F5-TTS: a fully non-autoregressive text-to-speech model that fills in speech with flow matching, using a Diffusion Transformer and ConvNeXt V2 text refinement over characters padded to the speech length.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → FlowMatchingTtsModelBase<T> → F5TTS<T>

Implements: IAcousticModel<T>

For Beginners

F5-TTS learns to fill in a masked part of a spectrogram given the text; to speak, it fills in everything after an optional voice prompt.

How It Works

References: “F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching” (Chen et al., 2024) and its reference implementation (SWivid/F5-TTS) for what the paper leaves unstated.

The characters, shifted so 0 is the filler token and padded to the frame count, are embedded, given an absolute sinusoidal position and refined by ConvNeXt V2 blocks (§3.2). They are concatenated with the noisy mel and the masked mel condition, projected and given a convolutional position embedding, then run through DiT blocks whose adaLN-zero is conditioned on the flow step, with rotary self-attention; a final adaptive LayerNorm and a zero-initialized projection give the flow. Training and sampling are FlowMatchingTtsModelBase’s.

Constructors

Constructor Summary
F5TTS(NeuralNetworkArchitecture<T> architecture, F5TTSOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
F5TTS(NeuralNetworkArchitecture<T> architecture, string modelPath, F5TTSOptions options)

Properties

Property Summary
int FftSize { get; }
int HopSize { get; }
int MaxTextLength { get; }
int MelChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> TextToMel(string text) Generates a mel spectrogram from text (the paper pairs it with Vocos).