MelSpectrogram<T>

Models & Types · AiDotNet.Diffusion.Audio

Computes Mel spectrograms from audio signals.

For Beginners

Human hearing doesn’t perceive pitch linearly - we can tell the difference between 100Hz and 200Hz easily, but 10,000Hz and 10,100Hz sound almost the same to us. The Mel scale accounts for this.

A Mel spectrogram:

  1. Computes the power spectrogram using STFT
  2. Applies a bank of triangular filters on the Mel scale
  3. Takes the log (optional) to compress dynamic range

This representation is commonly used for:

  • Speech recognition (like Whisper)
  • Music generation (like Riffusion)
  • Audio classification
  • Speaker verification

Usage:

var melSpec = new MelSpectrogram<float>(
sampleRate: 44100,
nMels: 128,
nFft: 2048
);
var audioSignal = Tensor<float>.CreateRandom(44100); // one second at 44.1 kHz
var mel = melSpec.Forward(audioSignal);
// mel.Shape = [numFrames, nMels]

How It Works

The Mel spectrogram is a representation of audio that mimics human hearing. It applies the Mel scale, which spaces frequencies according to how humans perceive pitch rather than the physical frequency.

Constructors

Constructor Summary
MelSpectrogram(int sampleRate, int nMels, int nFft, int? hopLength, double fMin, double? fMax, IWindowFunction<T> windowFunction, bool logMel, double refDb, double minDb, double? logOffset) Initializes a new Mel spectrogram processor.

Properties

Property Summary
ShortTimeFourierTransform<T> STFT { get; } Gets the STFT parameters.
int NumMels { get; } Gets the number of Mel bins.
int SampleRate { get; } Gets the sample rate.

Methods

Method Summary
Tensor<T> DbToPower(Tensor<T> db) Converts dB spectrogram back to power.
Tensor<T> Forward(Tensor<T> signal) Computes the Mel spectrogram of an audio signal.
Tensor<T> FromPowerSpectrogram(Tensor<T> powerSpectrogram) Computes Mel spectrogram from a pre-computed power spectrogram.
Tensor<T> GetFilterbank() Gets the Mel filterbank matrix.
Tensor<T> InvertMelToMagnitude(Tensor<T> melSpec, bool? isDb) Inverts a Mel spectrogram to approximate magnitude spectrogram.
double HzToMel(double hz) Converts frequency in Hz to Mel scale.
double MelToHz(double mel) Converts frequency in Mels to Hz.
double[] GetMelCenterFrequencies() Computes the frequency (in Hz) for each Mel bin center.