CoMoSpeech<T>

Models & Types · AiDotNet.TextToSpeech.FlowDiffusion

CoMoSpeech: one-step speech synthesis with a consistency model distilled from an EDM diffusion teacher.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → CoMoSpeech<T>

Implements: IAcousticModel<T>, IEndToEndTts<T>

For Beginners

A slow diffusion “teacher” learns to clean noisy spectrograms over many steps; the “student” then learns to jump from any point of that cleaning path straight to the end, so it speaks in one step.

How It Works

References: “CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency Model” (Ye et al., ACM MM 2023) and its reference implementation (zhenye234/CoMoSpeech) for what the paper leaves unstated.

The encoder, duration predictor and monotonic alignment search are Grad-TTS’s, giving the prior μ. The denoiser is EDM-preconditioned (Eq. 8–9): D(x, t) = c_skip(t) x + c_out(t) F(c_in(t) x, μ, ln t / 4) with c_skip = σ_d² / ((t − ε)² + σ_d²), c_out = σ_d (t − ε) / √(σ_d² + t²), c_in = 1 / √(σ_d² + t²), and F is Grad-TTS’s U-Net. The teacher (Teacher) minimizes the duration loss, the prior loss and the EDM-weighted denoising loss λ(t) ‖D(x₀ + t n, t) − x₀‖², n ~ N(μ, I), ln t ~ N(−1.2, 1.2²) (Eq. 10), and samples with N Euler steps of dx = (x − D(x, t)) / t dt from t_N x_N, x_N ~ N(μ, I) (Algorithm 1). Distillation (Distillation, Eq. 11–12) freezes the encoder and a copy of the teacher, keeps an EMA target θ⁻, and minimizes ‖D_θ(x_{n+1}, t_{n+1}) − D_θ⁻(x̂_n, t_n)‖² where x̂_n is one teacher Euler step from x_{n+1}; the distilled model synthesizes in one step, D_θ(t_N x_N, t_N) (Eq. 13), or in several with Algorithm 2.

Constructors

Constructor Summary
CoMoSpeech(NeuralNetworkArchitecture<T> architecture, CoMoSpeechOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
CoMoSpeech(NeuralNetworkArchitecture<T> architecture, string modelPath, CoMoSpeechOptions options)

Properties

Property Summary
CoMoSpeechTrainingPhase CurrentPhase { get; set; } The training stage; BeginDistillation moves from the teacher to distillation.
int FftSize { get; }
int HiddenDim { get; }
int HopSize { get; }
int MaxTextLength { get; }
int MelChannels { get; }
int NumFlowSteps { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
Tensor<T> TextToMel(string text) Generates a mel spectrogram from text (the paper pairs it with HiFi-GAN).
void BeginDistillation() Starts consistency distillation (§3.2, §3.4): the student keeps the teacher’s weights (θ and θ⁻ are “initialized weights of CoMoSpeech inherited from the teacher model”), a frozen copy of the denoiser becomes the ODE solver φ, and from now…