CoMoSpeech<T>
Models & Types · AiDotNet.TextToSpeech.FlowDiffusion
CoMoSpeech: one-step speech synthesis with a consistency model distilled from an EDM diffusion teacher.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → CoMoSpeech<T>
Implements: IAcousticModel<T>, IEndToEndTts<T>
For Beginners
A slow diffusion “teacher” learns to clean noisy spectrograms over many steps; the “student” then learns to jump from any point of that cleaning path straight to the end, so it speaks in one step.
How It Works
References: “CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency Model” (Ye et al., ACM MM 2023) and its reference implementation (zhenye234/CoMoSpeech) for what the paper leaves unstated.
The encoder, duration predictor and monotonic alignment search are Grad-TTS’s, giving the prior μ. The denoiser is
EDM-preconditioned (Eq. 8–9): D(x, t) = c_skip(t) x + c_out(t) F(c_in(t) x, μ, ln t / 4) with
c_skip = σ_d² / ((t − ε)² + σ_d²), c_out = σ_d (t − ε) / √(σ_d² + t²), c_in = 1 / √(σ_d² + t²), and F
is Grad-TTS’s U-Net. The teacher (Teacher) minimizes the duration loss, the
prior loss and the EDM-weighted denoising loss λ(t) ‖D(x₀ + t n, t) − x₀‖², n ~ N(μ, I), ln t ~ N(−1.2, 1.2²)
(Eq. 10), and samples with N Euler steps of dx = (x − D(x, t)) / t dt from t_N x_N, x_N ~ N(μ, I)
(Algorithm 1). Distillation (Distillation, Eq. 11–12) freezes the encoder and a
copy of the teacher, keeps an EMA target θ⁻, and minimizes ‖D_θ(x_{n+1}, t_{n+1}) − D_θ⁻(x̂_n, t_n)‖² where x̂_n
is one teacher Euler step from x_{n+1}; the distilled model synthesizes in one step, D_θ(t_N x_N, t_N) (Eq. 13),
or in several with Algorithm 2.
Constructors
| Constructor | Summary |
|---|---|
CoMoSpeech(NeuralNetworkArchitecture<T> architecture, CoMoSpeechOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
CoMoSpeech(NeuralNetworkArchitecture<T> architecture, string modelPath, CoMoSpeechOptions options) |
Properties
| Property | Summary |
|---|---|
CoMoSpeechTrainingPhase CurrentPhase { get; set; } |
The training stage; BeginDistillation moves from the teacher to distillation. |
int FftSize { get; } |
|
int HiddenDim { get; } |
|
int HopSize { get; } |
|
int MaxTextLength { get; } |
|
int MelChannels { get; } |
|
int NumFlowSteps { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> TextToMel(string text) |
Generates a mel spectrogram from text (the paper pairs it with HiFi-GAN). |
void BeginDistillation() |
Starts consistency distillation (§3.2, §3.4): the student keeps the teacher’s weights (θ and θ⁻ are “initialized weights of CoMoSpeech inherited from the teacher model”), a frozen copy of the denoiser becomes the ODE solver φ, and from now… |