MultiBandMelGAN<T>

Models & Types · AiDotNet.TextToSpeech.Vocoders

Multi-band MelGAN: a MelGAN generator that predicts PQMF sub-bands of the waveform at a quarter of its rate, trained with full- and sub-band multi-resolution STFT losses and multi-scale discriminators.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VocoderBase<T> → GanVocoderBase<T> → MultiBandMelGAN<T>

For Beginners

Instead of generating every audio sample, the network generates four narrow frequency bands at a quarter of the sample rate and a fixed filter bank merges them, which makes it several times faster.

How It Works

References: “Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to-Speech” (Yang et al., SLT 2021) and kan-bayashi/ParallelWaveGAN for what the paper leaves unstated.

The generator predicts four sub-band signals; the PQMF synthesis filter merges them into the full-band waveform the discriminators see (§2.2, §3.2). The generator first trains alone for 200k steps on L_mr_stft = ½ (L_full + L_sub) (Eq. 9; sub-band targets from the PQMF analysis filter), then the discriminators train on the LSGAN loss (Eq. 1) and the generator on λ_adv Σ_k (D_k(G(s)) − 1)² + L_mr_stft (Eq. 8). Both use Adam at 1e-4, halved every 100k steps to 1e-6.

Constructors

Constructor Summary
MultiBandMelGAN(NeuralNetworkArchitecture<T> architecture, MultiBandMelGANOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a trainable Multi-band MelGAN.
MultiBandMelGAN(NeuralNetworkArchitecture<T> architecture, string modelPath, MultiBandMelGANOptions options) Creates a trainable Multi-band MelGAN.

Properties

Property Summary
int UpsampleFactor { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()