DiffWave<T>
Models & Types · AiDotNet.TextToSpeech.Vocoders
DiffWave: a versatile diffusion model for audio synthesis — a non-autoregressive noise predictor built from bidirectional dilated convolutions, conditioned on the mel spectrogram, sampled by the reverse diffusion process.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VocoderBase<T> → SegmentVocoderBase<T> → DiffusionVocoderBase<T> → DiffWave<T>
For Beginners
DiffWave learns to remove noise from audio a little at a time; to synthesize, it starts from noise and removes it step by step, guided by the spectrogram.
How It Works
References: “DiffWave: A Versatile Diffusion Model for Audio Synthesis” (Kong et al., ICLR 2021) and lmnt-com/diffwave for what the paper leaves unstated.
Training (Algorithm 1): a step t uniform over the schedule, x_t = √ᾱ_t x₀ + √(1 − ᾱ_t) ε, and
‖ε − ε_θ(x_t, t, mel)‖². Synthesis (Algorithm 2) runs the reverse process over the training schedule, or over
the six-step fast schedule (App. B, Algorithm 3), whose steps are aligned to fractional training steps by their noise
levels with the step embedding interpolated between neighbours.
Example
// A vocoder: mel spectrogram in, waveform out.
var vocoder = new DiffWave<float>(architecture);
var speech = vocoder.MelToWaveform(mel);
// Unconditional and class-conditional generation (§5.2, §5.3).
var digits = new DiffWave<float>(architecture, DiffWaveOptions.ClassConditional(numClasses: 10));
digits.Train(new TtsTrainingSample<float> { Tokens = none, Audio = clip, ClassLabel = 7 });
var seven = digits.Generate(classLabel: 7);
Constructors
| Constructor | Summary |
|---|---|
DiffWave(NeuralNetworkArchitecture<T> architecture, DiffWaveOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a trainable DiffWave. |
DiffWave(NeuralNetworkArchitecture<T> architecture, string modelPath, DiffWaveOptions options) |
Creates a trainable DiffWave. |
Properties
| Property | Summary |
|---|---|
int UpsampleFactor { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> Generate(int? classLabel) |
Generates an utterance of UtteranceSamples samples by the reverse process (Algorithm 2): unconditionally, or of class classLabel for a class-conditional model. |