AudioLDM2Model<T>

Models & Types · AiDotNet.Diffusion.Audio

AudioLDM 2 - Enhanced Audio Latent Diffusion Model with dual text encoders.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → AudioDiffusionModelBase<T> → AudioLDM2Model<T>

For Beginners

AudioLDM 2 generates higher-quality audio than AudioLDM 1:

Example prompts:

  • “A symphony orchestra playing a dramatic crescendo” -> orchestral music
  • “Footsteps on gravel with birds chirping” -> detailed soundscape
  • “Electric guitar riff with heavy distortion” -> rock music

The dual encoder architecture means:

  • CLAP encoder understands audio concepts (instrument sounds, effects)
  • T5/GPT-2 encoder understands language (descriptions, context)
  • Combined, they produce audio that matches both sound and meaning

How It Works

AudioLDM 2 is an improved version of AudioLDM with significant architectural enhancements for better text-to-audio and text-to-music generation. Key improvements include:

  1. Dual Text Encoders: Combines CLAP (audio-text) and T5/GPT-2 (language) embeddings
  2. Larger Architecture: 384 base channels vs 256 in AudioLDM 1
  3. Higher Resolution: 128 mel channels vs 64 for better audio quality
  4. Improved Music Generation: Better temporal coherence and musical structure
  5. Longer Duration Support: Up to 30 seconds of audio generation

Technical specifications:

  • Sample rate: 16 kHz (speech/effects) or 48 kHz (high-quality music)
  • Latent channels: 8
  • Mel channels: 128 (double AudioLDM 1)
  • Base channels: 384 (1.5x AudioLDM 1)
  • Context dimension: 1024 (combined encoder output)
  • Duration: Up to 30 seconds
  • Guidance scale: 3.0-6.0 typical

Example

// Create an AudioLDM 2 model
var audioLDM2 = new AudioLDM2Model<float>();
            
// Generate high-quality music
var music = audioLDM2.GenerateMusic(
    prompt: "Cinematic orchestral music with dramatic strings and timpani",
    durationSeconds: 20.0,
    numInferenceSteps: 200,
    guidanceScale: 4.5);
            
// Generate complex sound effects
var soundscape = audioLDM2.GenerateAudio(
    prompt: "A busy city street with traffic, horns, and people talking",
    durationSeconds: 15.0,
    numInferenceSteps: 150,
    guidanceScale: 4.0);

Constructors

Constructor Summary
AudioLDM2Model(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> unet, AudioVAE<T> audioVAE, IConditioningModule<T> clapConditioner, IConditioningModule<T> languageConditioner, AudioLDM2Variant variant, int sampleRate, double defaultDurationSeconds, int? seed) Initializes a new AudioLDM 2 model with full customization support.

Properties

Property Summary
AudioLDM2Variant Variant { get; } Gets the model variant.
AudioVAE<T> AudioVAE { get; } Gets the AudioVAE for direct access.
IConditioningModule<T> Conditioner { get; }
IConditioningModule<T> LanguageConditioner { get; } Gets the secondary language conditioning module.
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> VAE { get; }
bool SupportsAudioToAudio { get; }
bool SupportsTextToAudio { get; }
bool SupportsTextToMusic { get; }
bool SupportsTextToSpeech { get; }
int LatentChannels { get; }

Methods

Method Summary
List<Tensor<T>> GenerateVariations(Tensor<T> inputAudio, int numVariations, double variationStrength, int? seed) Generates audio variations with enhanced diversity.
List<Tensor<T>> InterpolateAudio(Tensor<T> audio1, Tensor<T> audio2, int numSteps) Interpolates between two audio samples in latent space.
Tensor<T> GenerateAudio(string prompt, string negativePrompt, double? durationSeconds, int numInferenceSteps, double guidanceScale, int? seed) Generates audio from a text prompt using dual encoders.
Tensor<T> GenerateMusic(string prompt, string negativePrompt, double? durationSeconds, int numInferenceSteps, double guidanceScale, int? seed) Generates music from a text prompt with enhanced musical understanding.
Tensor<T> TransformAudio(Tensor<T> inputAudio, string prompt, string negativePrompt, double strength, int numInferenceSteps, double guidanceScale, int? seed) Transforms audio based on a text prompt (audio-to-audio).

Fields

Field Summary
double AUDIOLDM2_MAX_DURATION Maximum supported duration in seconds.
int AUDIOLDM2_BASE_CHANNELS AudioLDM 2 U-Net base channels (larger than AudioLDM 1).
int AUDIOLDM2_CONTEXT_DIM Combined context dimension from dual encoders.
int AUDIOLDM2_LATENT_CHANNELS AudioLDM 2 latent space channels.
int AUDIOLDM2_MEL_CHANNELS AudioLDM 2 mel spectrogram channels (increased from 64 to 128).
int AUDIOLDM2_SAMPLE_RATE AudioLDM 2 default sample rate for high-quality audio.