AudioVAE<T>
Models & Types · AiDotNet.Diffusion.VAE
AudioLDM’s variational autoencoder: the latent-diffusion KL autoencoder applied to a log-mel spectrogram as a one-channel image.
Inheritance: VAEModelBase<T> → AudioVAE<T>
For Beginners
Raw audio is far too long for a diffusion model, so AudioLDM works on a compressed picture of it. The audio becomes a mel spectrogram (time on one axis, pitch bands on the other), and this VAE squeezes that picture 4x along both axes into 8 latent channels. The diffusion model works there, and the decoder turns its result back into a spectrogram.
How It Works
References:
- Paper: “AudioLDM: Text-to-Audio Generation with Latent Diffusion Models” (Liu et al., ICML 2023), Sec. 3.2
- Implementation: diffusers
AutoencoderKLas thecvssp/audioldm-s-full-v2andcvssp/audioldm2
pipelines configure it, which this reproduces (tools/reference-data/audioldm_vae_reference.py);
LoadDiffusersWeights loads its state dict.
Architecture (diffusers’ AutoencoderKL): a 3x3 conv in; one stage of residual blocks per width in
blockOutChannels, each stage but the last halving both axes; a middle block of residual block, single-head
self-attention and residual block; GroupNorm, SiLU and a 3x3 conv to the posterior’s mean and log-variance; a 1x1
quant conv. The decoder mirrors it (a 1x1 post-quant conv first, one more residual block per stage, nearest 2x
upsampling). A spectrogram is [batch, 1, time frames, mel bins], as AudioLDM lays it out.
Example
var vae = new AudioVAE<float>();
var waveform = Tensor<float>.CreateRandom(new[] { 1, 16000 }); // one second at 16 kHz
var mel = vae.AudioToMelSpectrogram(waveform); // [1, 1, frames, 64]
var latent = vae.Encode(mel, sampleMode: false); // [1, 8, frames / 4, 16]
Constructors
| Constructor | Summary |
|---|---|
AudioVAE(int inputChannels, int latentChannels, int[] blockOutChannels, int layersPerBlock, int normGroups, int melBins, double latentScaleFactor, ILossFunction<T> lossFunction, int? seed) |
Initializes an AudioVAE in AudioLDM’s configuration (diffusers AutoencoderKL, cvssp/audioldm-s-full-v2). |
Properties
| Property | Summary |
|---|---|
double LatentScaleFactor { get; } |
|
int DownsampleFactor { get; } |
|
int InputChannels { get; } |
|
int LatentChannels { get; } |
|
int MelBins { get; } |
Gets the number of mel bins the spectrogram helpers produce and read. |
Methods
| Method | Summary |
|---|---|
IReadOnlyCollection<string> LoadDiffusersWeights(Func<string, int[], double[]> read) |
Loads a diffusers AutoencoderKL state dict straight from source into the weights. |
Tensor<T> AudioToMelSpectrogram(Tensor<T> waveform, int sampleRate, int hopLength, int fftSize) |
Converts a waveform to the log-mel spectrogram this VAE reads, [batch, 1, time frames, mel bins]. |
Tensor<T> Decode(Tensor<T> latent) |
|
Tensor<T> Encode(Tensor<T> input, bool sampleMode) |
|
Tensor<T> MelSpectrogramToAudio(Tensor<T> melSpectrogram, int sampleRate, int hopLength) |
Converts a log-mel spectrogram, [batch, 1, time frames, mel bins], back to a waveform by inverting the mel filterbank and reconstructing the phase with Griffin-Lim (AudioLDM itself uses a HiFi-GAN vocoder). |
ValueTuple<Tensor<T>, Tensor<T>> EncodeWithDistribution(Tensor<T> input) |
Fields
| Field | Summary |
|---|---|
double AudioLdm2LatentScale |
The scaling factor of AudioLDM 2’s VAE (cvssp/audioldm2). |
double AudioLdmLatentScale |
The scaling factor of AudioLDM’s VAE (cvssp/audioldm-s-full-v2). |