VoiceFlow<T>

Models & Types · AiDotNet.TextToSpeech.FlowDiffusion

VoiceFlow: a text-to-speech acoustic model that generates mel spectrograms with conditional flow matching and straightens its sampling trajectory with rectified flow.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VoiceFlow<T>

Implements: IAcousticModel<T>, IEndToEndTts<T>

For Beginners

VoiceFlow learns to move random noise in a straight line to a spectrogram; training a second time on its own noise/output pairs makes the lines straighter, so very few steps are needed.

How It Works

References: “VoiceFlow: Efficient Text-to-Speech with Rectified Flow Matching” (Guo et al., ICASSP 2024) and its reference implementation (cantabile-kwok/VoiceFlow-TTS) for what the paper leaves unstated.

The text encoder and duration predictor are Grad-TTS’s; durations come from a forced alignment (§3.1, §4.1) and duplicate the encoder output into the frame-level condition y. A Grad-TTS U-Net u_θ(x_t, y, t) estimates the vector field of the path p_t(x | x₀, x₁) = N(t x₁ + (1 − t) x₀, σ² I), x₀ ~ N(0, I), whose target is the straight line x₁ − x₀ (Eq. 4–5); the loss is L_FM + L_dur. Sampling solves dx = u_θ dt with N Euler steps from t = 0 to 1 (Eq. 6). Flow rectification (§3.2, Algorithm 1) samples a noise x₀′ per utterance, solves the ODE with the ground-truth durations to obtain x̂₁ (GenerateRectificationPair), and trains again on the fixed pair (TrainRectified, Eq. 7).

Constructors

Constructor Summary
VoiceFlow(NeuralNetworkArchitecture<T> architecture, VoiceFlowOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
VoiceFlow(NeuralNetworkArchitecture<T> architecture, string modelPath, VoiceFlowOptions options)

Properties

Property Summary
int FftSize { get; }
int HiddenDim { get; }
int HopSize { get; }
int MaxTextLength { get; }
int MelChannels { get; }
int NumFlowSteps { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
T TrainRectified(TtsTrainingSample<T> sample, Tensor<T> noise) Flow rectification, training (Eq.
Tensor<T> TextToMel(string text) Generates a mel spectrogram from text (the paper pairs it with HiFi-GAN).
ValueTuple<Tensor<T>, Tensor<T>> GenerateRectificationPair(TtsTrainingSample<T> sample, int? steps) Flow rectification, data generation (Algorithm 1 lines 10–11): draws a noise x₀′ for the utterance and solves the ODE from it with the ground-truth durations, returning the pair (x₀′, x̂₁), both [frames, mel].