Vocos<T>

Models & Types · AiDotNet.TextToSpeech.Vocoders

Vocos: a GAN vocoder that generates Fourier spectral coefficients — magnitude and phase — at frame rate with a ConvNeXt backbone and turns them into audio with the inverse STFT.

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → VocoderBase<T> → GanVocoderBase<T> → Vocos<T>

For Beginners

Rather than building audio sample by sample, Vocos predicts how loud and in what phase each frequency is in every frame and lets an inverse Fourier transform assemble the sound, which is very fast.

How It Works

References: “Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis” (Siuzdak, ICLR 2024) and gemelo-ai/vocos for what the paper leaves unstated.

The head’s magnitude is exp(m) and its phase the point (cos p, sin p) on the unit circle, wrapping implicitly (§3.2). The discriminators train on the hinge loss 1/K Σ_k max(0, 1 − D_k(x)) + max(0, 1 + D_k(x̂)) and the generator on 1/K Σ_k max(0, 1 − D_k(x̂)) plus feature matching and the mel L1 (§3.3), the multi-resolution discriminator’s terms weighted 0.1 and the mel term 45 as the reference trains it. AdamW (0.9, 0.999) at 2e-4 with a cosine decay (§4.1).

Constructors

Constructor Summary
Vocos(NeuralNetworkArchitecture<T> architecture, VocosOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a trainable Vocos.
Vocos(NeuralNetworkArchitecture<T> architecture, string modelPath, VocosOptions options) Creates a trainable Vocos.

Properties

Property Summary
int UpsampleFactor { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()