UnivNet<T>

Models & Types · AiDotNet.TextToSpeech.Vocoders

UnivNet: a GAN vocoder whose generator uses location-variable convolutions with kernels predicted from the mel spectrogram, trained against multi-resolution spectrogram and multi-period waveform discriminators.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VocoderBase<T> → GanVocoderBase<T> → UnivNet<T>

For Beginners

Every short stretch of the spectrogram designs its own small filter, which the network uses to shape noise into the matching stretch of audio; critics looking at spectrograms of several resolutions keep the result sharp.

How It Works

References: “UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation” (Jang et al., Interspeech 2021) and maum-ai/univnet for what the paper leaves unstated.

The generator shapes Gaussian noise into a waveform through three LVC residual stacks (§3.1). It first trains alone for 200k steps on λ · L_aux, the multi-resolution STFT loss (§3.3, §4.3); afterwards each step updates the generator on λ · L_aux plus the LSGAN generator loss averaged over the three resolution and five period discriminators (§3.2), and the discriminators on the averaged LSGAN loss against that same generation. Adam (β = 0.5, 0.9) at 1e-4.

Constructors

Constructor Summary
UnivNet(NeuralNetworkArchitecture<T> architecture, UnivNetOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a trainable UnivNet.
UnivNet(NeuralNetworkArchitecture<T> architecture, string modelPath, UnivNetOptions options) Creates a trainable UnivNet.

Properties

Property Summary
int UpsampleFactor { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()