DiffWave<T>

Models & Types · AiDotNet.TextToSpeech.Vocoders

DiffWave: a versatile diffusion model for audio synthesis — a non-autoregressive noise predictor built from bidirectional dilated convolutions, conditioned on the mel spectrogram, sampled by the reverse diffusion process.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VocoderBase<T> → SegmentVocoderBase<T> → DiffusionVocoderBase<T> → DiffWave<T>

For Beginners

DiffWave learns to remove noise from audio a little at a time; to synthesize, it starts from noise and removes it step by step, guided by the spectrogram.

How It Works

References: “DiffWave: A Versatile Diffusion Model for Audio Synthesis” (Kong et al., ICLR 2021) and lmnt-com/diffwave for what the paper leaves unstated.

Training (Algorithm 1): a step t uniform over the schedule, x_t = √ᾱ_t x₀ + √(1 − ᾱ_t) ε, and ‖ε − ε_θ(x_t, t, mel)‖². Synthesis (Algorithm 2) runs the reverse process over the training schedule, or over the six-step fast schedule (App. B, Algorithm 3), whose steps are aligned to fractional training steps by their noise levels with the step embedding interpolated between neighbours.

Example

// A vocoder: mel spectrogram in, waveform out.
var vocoder = new DiffWave<float>(architecture);
var speech = vocoder.MelToWaveform(mel);
// Unconditional and class-conditional generation (§5.2, §5.3).
var digits = new DiffWave<float>(architecture, DiffWaveOptions.ClassConditional(numClasses: 10));
digits.Train(new TtsTrainingSample<float> { Tokens = none, Audio = clip, ClassLabel = 7 });
var seven = digits.Generate(classLabel: 7);

Constructors

Constructor Summary
DiffWave(NeuralNetworkArchitecture<T> architecture, DiffWaveOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a trainable DiffWave.
DiffWave(NeuralNetworkArchitecture<T> architecture, string modelPath, DiffWaveOptions options) Creates a trainable DiffWave.

Properties

Property Summary
int UpsampleFactor { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Tensor<T> Generate(int? classLabel) Generates an utterance of UtteranceSamples samples by the reverse process (Algorithm 2): unconditionally, or of class classLabel for a class-conditional model.