ISTFTNet<T>

Models & Types · AiDotNet.TextToSpeech.Vocoders

iSTFTNet: a fast, lightweight mel-spectrogram vocoder — HiFi-GAN with its output-side upsampling replaced by an inverse STFT of a small magnitude and phase spectrogram it predicts.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VocoderBase<T> → GanVocoderBase<T> → ISTFTNet<T>

For Beginners

Instead of building every audio sample with neural layers, the network predicts a tiny spectrogram and a fixed mathematical transform turns it into sound, which is much faster.

How It Works

References: “iSTFTNet: Fast and Lightweight Mel-Spectrogram Vocoder Incorporating Inverse Short-Time Fourier Transform” (Kaneko et al., ICASSP 2022).

After two ×8 upsamplings the output convolution gives (f/2 + 1) × 2 channels; an exponential makes the first half a linear magnitude and a sine makes the second half the phase (§3.3), and iSTFT(16, 4, 16) turns them into the waveform (Eq. 1). Training is HiFi-GAN’s — its discriminators, LSGAN, feature matching (×2) and mel L1 (×45) — with Adam (β = 0.5, 0.9) at 2e-4 (§4.1).

Constructors

Constructor Summary
ISTFTNet(NeuralNetworkArchitecture<T> architecture, ISTFTNetOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a trainable iSTFTNet.
ISTFTNet(NeuralNetworkArchitecture<T> architecture, string modelPath, ISTFTNetOptions options) Creates a trainable iSTFTNet.

Properties

Property Summary
int UpsampleFactor { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()