MelSpectrogram<T>
Models & Types · AiDotNet.Diffusion.Audio
Computes Mel spectrograms from audio signals.
For Beginners
Human hearing doesn’t perceive pitch linearly - we can tell the difference between 100Hz and 200Hz easily, but 10,000Hz and 10,100Hz sound almost the same to us. The Mel scale accounts for this.
A Mel spectrogram:
- Computes the power spectrogram using STFT
- Applies a bank of triangular filters on the Mel scale
- Takes the log (optional) to compress dynamic range
This representation is commonly used for:
- Speech recognition (like Whisper)
- Music generation (like Riffusion)
- Audio classification
- Speaker verification
Usage:
var melSpec = new MelSpectrogram<float>(
sampleRate: 44100,
nMels: 128,
nFft: 2048
);
var audioSignal = Tensor<float>.CreateRandom(44100); // one second at 44.1 kHz
var mel = melSpec.Forward(audioSignal);
// mel.Shape = [numFrames, nMels]
How It Works
The Mel spectrogram is a representation of audio that mimics human hearing. It applies the Mel scale, which spaces frequencies according to how humans perceive pitch rather than the physical frequency.
Constructors
| Constructor | Summary |
|---|---|
MelSpectrogram(int sampleRate, int nMels, int nFft, int? hopLength, double fMin, double? fMax, IWindowFunction<T> windowFunction, bool logMel, double refDb, double minDb, double? logOffset) |
Initializes a new Mel spectrogram processor. |
Properties
| Property | Summary |
|---|---|
ShortTimeFourierTransform<T> STFT { get; } |
Gets the STFT parameters. |
int NumMels { get; } |
Gets the number of Mel bins. |
int SampleRate { get; } |
Gets the sample rate. |
Methods
| Method | Summary |
|---|---|
Tensor<T> DbToPower(Tensor<T> db) |
Converts dB spectrogram back to power. |
Tensor<T> Forward(Tensor<T> signal) |
Computes the Mel spectrogram of an audio signal. |
Tensor<T> FromPowerSpectrogram(Tensor<T> powerSpectrogram) |
Computes Mel spectrogram from a pre-computed power spectrogram. |
Tensor<T> GetFilterbank() |
Gets the Mel filterbank matrix. |
Tensor<T> InvertMelToMagnitude(Tensor<T> melSpec, bool? isDb) |
Inverts a Mel spectrogram to approximate magnitude spectrogram. |
double HzToMel(double hz) |
Converts frequency in Hz to Mel scale. |
double MelToHz(double mel) |
Converts frequency in Mels to Hz. |
double[] GetMelCenterFrequencies() |
Computes the frequency (in Hz) for each Mel bin center. |