MultiResolutionStftLoss<T>

Models & Types · AiDotNet.LossFunctions

Multi-resolution STFT loss: spectral convergence plus log-magnitude distance, summed over several STFT resolutions.

Inheritance: LossFunctionBase<T> → MultiResolutionStftLoss<T>

For Beginners

Comparing two sounds sample by sample is a poor measure of whether they sound alike — shifting a waveform slightly changes every sample while sounding identical. Comparing spectrograms instead asks “does it have the same energy at the same frequencies at roughly the same times”, which is much closer to hearing. Doing it at several zoom levels at once matters because a short window resolves sharp clicks and a long window resolves slow decays, and a room impulse response has both.

How It Works

For each resolution r the loss adds two terms over the magnitude spectrograms of the prediction and the target:

spectral convergence L_SC = || |STFT(h)| - |STFT(h^)| ||_F / || |STFT(h)| ||_F, and log-magnitude L_SM = (1/N) || log|STFT(h)| - log|STFT(h^)| ||_1.

This is the objective FiNS trains with (Steinmetz, Ithapu and Calamia, WASPAA 2021, arXiv:2107.07503), where the paper reports the multi-resolution STFT loss ALONE produced the best results, at R = 4 resolutions with frame sizes 64, 512, 2048 and 8192. It is also the standard Parallel WaveGAN objective (Yamamoto et al. 2020), so it is reusable by any waveform model, not only FiNS.

Why the transform is hand-rolled.IEngine.STFT returns magnitude and phase through out parameters, which places it outside the autodiff graph — the same reason APNet2GeneratorLoss cannot use it for its STFT-consistency term. So each resolution keeps constant cosine/sine DFT bases and frames the signal with differentiable tensor ops, giving a magnitude that gradients actually flow through.

Example

using AiDotNet.LossFunctions;
using AiDotNet.Tensors.LinearAlgebra;

var loss = new MultiResolutionStftLoss<float>();
var predicted = new Vector<float>(new[] { 0.9f, 0.2f, 0.7f });
var actual = new Vector<float>(new[] { 1.0f, 0.0f, 1.0f });

float value = loss.CalculateLoss(predicted, actual);
Console.WriteLine($"MultiResolutionStftLoss = {value:F4}");

Constructors

Constructor Summary
MultiResolutionStftLoss(int[] frameSizes, double epsilon) Creates the loss over the given STFT frame sizes.

Methods

Method Summary
T CalculateLoss(Vector<T> predicted, Vector<T> actual)
Tensor<T> ComputeTapeLoss(Tensor<T> predicted, Tensor<T> target)