AudioProcessor<T>
Models & Types · AiDotNet.Diffusion.Audio
Complete audio processing pipeline for diffusion-based audio generation.
For Beginners
This is your one-stop shop for working with audio in diffusion models. It handles:
- Converting audio waveforms to spectrograms (for training/conditioning)
- Converting spectrograms back to audio (for generation)
- Normalizing and denormalizing spectrograms
Typical workflow for Riffusion-style generation:
var processor = new AudioProcessor<float>(sampleRate: 44100);
// Encode reference audio to latent space (via spectrogram)
var referenceAudio = Tensor<float>.CreateRandom(44100); // one second at 44.1 kHz
var spectrogram = processor.AudioToSpectrogram(referenceAudio);
var normalized = processor.NormalizeSpectrogram(spectrogram);
// ... diffusion model generates new spectrogram ...
var generatedSpec = normalized;
// Decode generated spectrogram back to audio
var denormalized = processor.DenormalizeSpectrogram(generatedSpec);
var audio = processor.SpectrogramToAudio(denormalized);
How It Works
This class combines STFT, Mel spectrogram, and Griffin-Lim into a unified pipeline for audio analysis and synthesis. It’s designed for use with diffusion models like Riffusion that generate spectrograms.
Constructors
| Constructor | Summary |
|---|---|
AudioProcessor(int sampleRate, int nFft, int hopLength, int nMels, double fMin, double? fMax, double minDb, double maxDb, int griffinLimIterations) |
Initializes a new audio processor with Riffusion-compatible defaults. |
Properties
| Property | Summary |
|---|---|
GriffinLim<T> GriffinLim { get; } |
Gets the Griffin-Lim processor. |
MelSpectrogram<T> MelSpectrogram { get; } |
Gets the Mel spectrogram processor. |
ShortTimeFourierTransform<T> STFT { get; } |
Gets the STFT processor. |
int HopLength { get; } |
Gets the hop length. |
int NFft { get; } |
Gets the FFT size. |
int NumMels { get; } |
Gets the number of Mel bins. |
int SampleRate { get; } |
Gets the sample rate. |
Methods
| Method | Summary |
|---|---|
Tensor<T> AudioToImageSpectrogram(Tensor<T> audio, int targetWidth, int targetHeight) |
Creates a spectrogram suitable for image-based diffusion models. |
Tensor<T> AudioToSpectrogram(Tensor<T> audio) |
Converts audio waveform to a normalized Mel spectrogram. |
Tensor<T> DenormalizeSpectrogram(Tensor<T> normalizedSpectrogram) |
Denormalizes a [0, 1] spectrogram back to dB. |
Tensor<T> NormalizeAudio(Tensor<T> audio, double targetPeak) |
Normalizes audio to a peak amplitude. |
Tensor<T> NormalizeSpectrogram(Tensor<T> dbSpectrogram) |
Normalizes a dB spectrogram to [0, 1] range. |
Tensor<T> PadOrTruncate(Tensor<T> audio, int targetLength) |
Pads or truncates audio to a specific length. |
Tensor<T> PadOrTruncate(Tensor<T> audio, int targetLength, T padValue) |
Pads or truncates audio to a specific length. |
Tensor<T> SpectrogramToAudio(Tensor<T> spectrogram, int? length) |
Converts a normalized spectrogram back to audio. |
double FramesToDuration(int numFrames) |
Computes the duration of audio from spectrogram dimensions. |
double[] GetMelFrequencyAxis() |
Gets the frequency axis values for a Mel spectrogram. |
double[] GetTimeAxis(int numFrames) |
Gets the time axis values for a spectrogram. |
int DurationToFrames(double durationSeconds) |
Computes the number of frames for a given duration. |
int DurationToSamples(double durationSeconds) |
Computes the number of samples for a given duration. |