MultiResolutionStftLoss<T>
Models & Types · AiDotNet.LossFunctions
Multi-resolution STFT loss: spectral convergence plus log-magnitude distance, summed over several STFT resolutions.
Inheritance: LossFunctionBase<T> → MultiResolutionStftLoss<T>
For Beginners
Comparing two sounds sample by sample is a poor measure of whether they sound alike — shifting a waveform slightly changes every sample while sounding identical. Comparing spectrograms instead asks “does it have the same energy at the same frequencies at roughly the same times”, which is much closer to hearing. Doing it at several zoom levels at once matters because a short window resolves sharp clicks and a long window resolves slow decays, and a room impulse response has both.
How It Works
For each resolution r the loss adds two terms over the magnitude spectrograms of the prediction and the target:
spectral convergence L_SC = || |STFT(h)| - |STFT(h^)| ||_F / || |STFT(h)| ||_F, and
log-magnitude L_SM = (1/N) || log|STFT(h)| - log|STFT(h^)| ||_1.
This is the objective FiNS trains with (Steinmetz, Ithapu and Calamia, WASPAA 2021, arXiv:2107.07503), where the paper reports the multi-resolution STFT loss ALONE produced the best results, at R = 4 resolutions with frame sizes 64, 512, 2048 and 8192. It is also the standard Parallel WaveGAN objective (Yamamoto et al. 2020), so it is reusable by any waveform model, not only FiNS.
Why the transform is hand-rolled.IEngine.STFT returns magnitude and phase through
out parameters, which places it outside the autodiff graph — the same reason
APNet2GeneratorLoss cannot use it for its STFT-consistency term. So each
resolution keeps constant cosine/sine DFT bases and frames the signal with differentiable tensor
ops, giving a magnitude that gradients actually flow through.
Example
using AiDotNet.LossFunctions;
using AiDotNet.Tensors.LinearAlgebra;
var loss = new MultiResolutionStftLoss<float>();
var predicted = new Vector<float>(new[] { 0.9f, 0.2f, 0.7f });
var actual = new Vector<float>(new[] { 1.0f, 0.0f, 1.0f });
float value = loss.CalculateLoss(predicted, actual);
Console.WriteLine($"MultiResolutionStftLoss = {value:F4}");
Constructors
| Constructor | Summary |
|---|---|
MultiResolutionStftLoss(int[] frameSizes, double epsilon) |
Creates the loss over the given STFT frame sizes. |
Methods
| Method | Summary |
|---|---|
T CalculateLoss(Vector<T> predicted, Vector<T> actual) |
|
Tensor<T> ComputeTapeLoss(Tensor<T> predicted, Tensor<T> target) |