AudioProcessor<T>

Models & Types · AiDotNet.Diffusion.Audio

Complete audio processing pipeline for diffusion-based audio generation.

For Beginners

This is your one-stop shop for working with audio in diffusion models. It handles:

  • Converting audio waveforms to spectrograms (for training/conditioning)
  • Converting spectrograms back to audio (for generation)
  • Normalizing and denormalizing spectrograms

Typical workflow for Riffusion-style generation:

var processor = new AudioProcessor<float>(sampleRate: 44100);

// Encode reference audio to latent space (via spectrogram)
var referenceAudio = Tensor<float>.CreateRandom(44100); // one second at 44.1 kHz
var spectrogram = processor.AudioToSpectrogram(referenceAudio);
var normalized = processor.NormalizeSpectrogram(spectrogram);

// ... diffusion model generates new spectrogram ...
var generatedSpec = normalized;

// Decode generated spectrogram back to audio
var denormalized = processor.DenormalizeSpectrogram(generatedSpec);
var audio = processor.SpectrogramToAudio(denormalized);

How It Works

This class combines STFT, Mel spectrogram, and Griffin-Lim into a unified pipeline for audio analysis and synthesis. It’s designed for use with diffusion models like Riffusion that generate spectrograms.

Constructors

Constructor Summary
AudioProcessor(int sampleRate, int nFft, int hopLength, int nMels, double fMin, double? fMax, double minDb, double maxDb, int griffinLimIterations) Initializes a new audio processor with Riffusion-compatible defaults.

Properties

Property Summary
GriffinLim<T> GriffinLim { get; } Gets the Griffin-Lim processor.
MelSpectrogram<T> MelSpectrogram { get; } Gets the Mel spectrogram processor.
ShortTimeFourierTransform<T> STFT { get; } Gets the STFT processor.
int HopLength { get; } Gets the hop length.
int NFft { get; } Gets the FFT size.
int NumMels { get; } Gets the number of Mel bins.
int SampleRate { get; } Gets the sample rate.

Methods

Method Summary
Tensor<T> AudioToImageSpectrogram(Tensor<T> audio, int targetWidth, int targetHeight) Creates a spectrogram suitable for image-based diffusion models.
Tensor<T> AudioToSpectrogram(Tensor<T> audio) Converts audio waveform to a normalized Mel spectrogram.
Tensor<T> DenormalizeSpectrogram(Tensor<T> normalizedSpectrogram) Denormalizes a [0, 1] spectrogram back to dB.
Tensor<T> NormalizeAudio(Tensor<T> audio, double targetPeak) Normalizes audio to a peak amplitude.
Tensor<T> NormalizeSpectrogram(Tensor<T> dbSpectrogram) Normalizes a dB spectrogram to [0, 1] range.
Tensor<T> PadOrTruncate(Tensor<T> audio, int targetLength) Pads or truncates audio to a specific length.
Tensor<T> PadOrTruncate(Tensor<T> audio, int targetLength, T padValue) Pads or truncates audio to a specific length.
Tensor<T> SpectrogramToAudio(Tensor<T> spectrogram, int? length) Converts a normalized spectrogram back to audio.
double FramesToDuration(int numFrames) Computes the duration of audio from spectrogram dimensions.
double[] GetMelFrequencyAxis() Gets the frequency axis values for a Mel spectrogram.
double[] GetTimeAxis(int numFrames) Gets the time axis values for a spectrogram.
int DurationToFrames(double durationSeconds) Computes the number of frames for a given duration.
int DurationToSamples(double durationSeconds) Computes the number of samples for a given duration.