AudioLDMModel<T>

Models & Types · AiDotNet.Diffusion.Audio

Audio Latent Diffusion Model (AudioLDM) for text-to-audio generation.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → AudioDiffusionModelBase<T> → AudioLDMModel<T>

For Beginners

AudioLDM lets you create sounds and music from text descriptions:

Example prompts:

  • “A dog barking in a park” -> generates dog barking sounds
  • “Rain falling on a window” -> generates rain sounds
  • “Jazz piano playing softly” -> generates jazz piano music

How it works:

  1. Text -> CLAP encoder -> text embedding (understands audio concepts)
  2. Text embedding guides diffusion in latent space
  3. Latent -> AudioVAE decoder -> mel spectrogram
  4. Mel spectrogram -> Vocoder -> audio waveform

Key features:

  • Text-to-audio: Generate sounds from descriptions
  • Audio-to-audio: Transform sounds while preserving some characteristics
  • Variable duration: Generate audio of different lengths
  • Classifier-free guidance: Control how closely to follow the prompt

How It Works

AudioLDM is a latent diffusion model specifically designed for audio generation. It works by generating mel spectrograms in latent space and then converting them to audio using a vocoder (like HiFi-GAN).

Technical specifications:

  • Sample rate: 16 kHz (standard for speech/effects) or 48 kHz (music)
  • Latent channels: 8
  • Mel channels: 64 (AudioLDM) or 128 (AudioLDM 2)
  • Duration: Typically 10 seconds, but configurable
  • Guidance scale: 2.5-5.0 typical

Example

// Create an AudioLDM model
var audioLDM = new AudioLDMModel<float>();
            
// Generate sound effects
var dogBark = audioLDM.GenerateFromText(
    prompt: "A dog barking excitedly",
    durationSeconds: 5.0,
    numInferenceSteps: 100,
    guidanceScale: 3.5);
            
// Generate music
var music = audioLDM.GenerateMusic(
    prompt: "Soft jazz piano with light drums",
    durationSeconds: 10.0,
    numInferenceSteps: 200,
    guidanceScale: 4.0);
            
// Save as audio file
// dogBark is the result tensor — write it out with your own encoder.

Constructors

Constructor Summary
AudioLDMModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> unet, AudioVAE<T> audioVAE, IConditioningModule<T> conditioner, int sampleRate, double defaultDurationSeconds, int melChannels, bool isVersion2, int? seed) Initializes a new AudioLDM model with full customization support.

Properties

Property Summary
AudioVAE<T> AudioVAE { get; } Gets the AudioVAE used for encoding/decoding.
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> VAE { get; }
bool IsVersion2 { get; } Gets whether this is AudioLDM version 2.
bool SupportsAudioToAudio { get; }
bool SupportsTextToAudio { get; }
bool SupportsTextToMusic { get; }
bool SupportsTextToSpeech { get; }
int LatentChannels { get; }

Methods

Method Summary
List<Tensor<T>> GenerateVariations(Tensor<T> inputAudio, int numVariations, double variationStrength, int? seed) Generates audio variations from an input audio.
Tensor<T> GenerateAudio(string prompt, string negativePrompt, double? durationSeconds, int numInferenceSteps, double guidanceScale, int? seed) Generates audio from a text prompt.
Tensor<T> GenerateMusic(string prompt, string negativePrompt, double? durationSeconds, int numInferenceSteps, double guidanceScale, int? seed) Generates music from a text prompt.
Tensor<T> TransformAudio(Tensor<T> inputAudio, string prompt, string negativePrompt, double strength, int numInferenceSteps, double guidanceScale, int? seed) Transforms audio based on a text prompt (audio-to-audio).

Fields

Field Summary
int AUDIOLDM_LATENT_CHANNELS Standard AudioLDM latent channels.
int AUDIOLDM_MEL_CHANNELS Standard AudioLDM mel channels.
int AUDIOLDM_SAMPLE_RATE Standard AudioLDM sample rate.