AudioVAE<T>

Models & Types · AiDotNet.Diffusion.VAE

AudioLDM’s variational autoencoder: the latent-diffusion KL autoencoder applied to a log-mel spectrogram as a one-channel image.

Inheritance: VAEModelBase<T> → AudioVAE<T>

For Beginners

Raw audio is far too long for a diffusion model, so AudioLDM works on a compressed picture of it. The audio becomes a mel spectrogram (time on one axis, pitch bands on the other), and this VAE squeezes that picture 4x along both axes into 8 latent channels. The diffusion model works there, and the decoder turns its result back into a spectrogram.

How It Works

References:

  • Paper: “AudioLDM: Text-to-Audio Generation with Latent Diffusion Models” (Liu et al., ICML 2023), Sec. 3.2
  • Implementation: diffusers AutoencoderKL as the cvssp/audioldm-s-full-v2 and cvssp/audioldm2

pipelines configure it, which this reproduces (tools/reference-data/audioldm_vae_reference.py); LoadDiffusersWeights loads its state dict.

Architecture (diffusers’ AutoencoderKL): a 3x3 conv in; one stage of residual blocks per width in blockOutChannels, each stage but the last halving both axes; a middle block of residual block, single-head self-attention and residual block; GroupNorm, SiLU and a 3x3 conv to the posterior’s mean and log-variance; a 1x1 quant conv. The decoder mirrors it (a 1x1 post-quant conv first, one more residual block per stage, nearest 2x upsampling). A spectrogram is [batch, 1, time frames, mel bins], as AudioLDM lays it out.

Example

var vae = new AudioVAE<float>();
var waveform = Tensor<float>.CreateRandom(new[] { 1, 16000 }); // one second at 16 kHz
var mel = vae.AudioToMelSpectrogram(waveform);   // [1, 1, frames, 64]
var latent = vae.Encode(mel, sampleMode: false); // [1, 8, frames / 4, 16]

Constructors

Constructor Summary
AudioVAE(int inputChannels, int latentChannels, int[] blockOutChannels, int layersPerBlock, int normGroups, int melBins, double latentScaleFactor, ILossFunction<T> lossFunction, int? seed) Initializes an AudioVAE in AudioLDM’s configuration (diffusers AutoencoderKL, cvssp/audioldm-s-full-v2).

Properties

Property Summary
double LatentScaleFactor { get; }
int DownsampleFactor { get; }
int InputChannels { get; }
int LatentChannels { get; }
int MelBins { get; } Gets the number of mel bins the spectrogram helpers produce and read.

Methods

Method Summary
IReadOnlyCollection<string> LoadDiffusersWeights(Func<string, int[], double[]> read) Loads a diffusers AutoencoderKL state dict straight from source into the weights.
Tensor<T> AudioToMelSpectrogram(Tensor<T> waveform, int sampleRate, int hopLength, int fftSize) Converts a waveform to the log-mel spectrogram this VAE reads, [batch, 1, time frames, mel bins].
Tensor<T> Decode(Tensor<T> latent)
Tensor<T> Encode(Tensor<T> input, bool sampleMode)
Tensor<T> MelSpectrogramToAudio(Tensor<T> melSpectrogram, int sampleRate, int hopLength) Converts a log-mel spectrogram, [batch, 1, time frames, mel bins], back to a waveform by inverting the mel filterbank and reconstructing the phase with Griffin-Lim (AudioLDM itself uses a HiFi-GAN vocoder).
ValueTuple<Tensor<T>, Tensor<T>> EncodeWithDistribution(Tensor<T> input)

Fields

Field Summary
double AudioLdm2LatentScale The scaling factor of AudioLDM 2’s VAE (cvssp/audioldm2).
double AudioLdmLatentScale The scaling factor of AudioLDM’s VAE (cvssp/audioldm-s-full-v2).