SegmentVocoderBase<T>

Base Classes · AiDotNet.TextToSpeech.Vocoders

The training and synthesis plumbing of vocoders trained by one objective on aligned (mel spectrogram, waveform) segments — the diffusion vocoders (DiffWave, WaveGrad, PriorGrad, FreGrad), the flow WaveGlow and the autoregressive WaveNet and WaveRNN: a model supplies its network, input features, segment length, objective and synthesis.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VocoderBase<T> → SegmentVocoderBase<T>

Known derived types: DiffusionVocoderBase<T>, WaveGlow<T>, WaveNet<T>, WaveRNN<T>

For Beginners

This is the shared machinery: cut a matching piece of spectrogram and audio, compute the model’s loss on it, update the weights; to make speech, run the model’s own generation procedure.

How It Works

Training takes a frame-aligned segment of SegmentSize samples (rounded down to whole frames) from a random frame, and one optimizer step on TrainingObjective. The random draws of a step (noise levels, noise, dropout of the objective) come from one seed per step, so the objective is a deterministic function of the parameters within the step. Synthesis runs with the model’s sampling seed, so it is repeatable.

Methods

Method Summary
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
Tensor<T> ComputeMel(Tensor<T> audio) The network’s input features for audio``[samples], [1, mel, frames].
Tensor<T> MelToWaveform(Tensor<T> melSpectrogram)
void Train(Tensor<T> input, Tensor<T> expectedOutput)