VoiceFlow<T>
Models & Types · AiDotNet.TextToSpeech.FlowDiffusion
VoiceFlow: a text-to-speech acoustic model that generates mel spectrograms with conditional flow matching and straightens its sampling trajectory with rectified flow.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VoiceFlow<T>
Implements: IAcousticModel<T>, IEndToEndTts<T>
For Beginners
VoiceFlow learns to move random noise in a straight line to a spectrogram; training a second time on its own noise/output pairs makes the lines straighter, so very few steps are needed.
How It Works
References: “VoiceFlow: Efficient Text-to-Speech with Rectified Flow Matching” (Guo et al., ICASSP 2024) and its reference implementation (cantabile-kwok/VoiceFlow-TTS) for what the paper leaves unstated.
The text encoder and duration predictor are Grad-TTS’s; durations come from a forced alignment (§3.1, §4.1) and
duplicate the encoder output into the frame-level condition y. A Grad-TTS U-Net u_θ(x_t, y, t) estimates the vector
field of the path p_t(x | x₀, x₁) = N(t x₁ + (1 − t) x₀, σ² I), x₀ ~ N(0, I), whose target is the straight
line x₁ − x₀ (Eq. 4–5); the loss is L_FM + L_dur. Sampling solves dx = u_θ dt with N Euler
steps from t = 0 to 1 (Eq. 6). Flow rectification (§3.2, Algorithm 1) samples a noise x₀′ per utterance, solves the
ODE with the ground-truth durations to obtain x̂₁ (GenerateRectificationPair), and trains again on the
fixed pair (TrainRectified, Eq. 7).
Constructors
| Constructor | Summary |
|---|---|
VoiceFlow(NeuralNetworkArchitecture<T> architecture, VoiceFlowOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
VoiceFlow(NeuralNetworkArchitecture<T> architecture, string modelPath, VoiceFlowOptions options) |
Properties
| Property | Summary |
|---|---|
int FftSize { get; } |
|
int HiddenDim { get; } |
|
int HopSize { get; } |
|
int MaxTextLength { get; } |
|
int MelChannels { get; } |
|
int NumFlowSteps { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
T TrainRectified(TtsTrainingSample<T> sample, Tensor<T> noise) |
Flow rectification, training (Eq. |
Tensor<T> TextToMel(string text) |
Generates a mel spectrogram from text (the paper pairs it with HiFi-GAN). |
ValueTuple<Tensor<T>, Tensor<T>> GenerateRectificationPair(TtsTrainingSample<T> sample, int? steps) |
Flow rectification, data generation (Algorithm 1 lines 10–11): draws a noise x₀′ for the utterance and solves the ODE from it with the ground-truth durations, returning the pair (x₀′, x̂₁), both [frames, mel]. |