AudioLDM2Model<T>
Models & Types · AiDotNet.Diffusion.Audio
AudioLDM 2 - Enhanced Audio Latent Diffusion Model with dual text encoders.
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → AudioDiffusionModelBase<T> → AudioLDM2Model<T>
For Beginners
AudioLDM 2 generates higher-quality audio than AudioLDM 1:
Example prompts:
- “A symphony orchestra playing a dramatic crescendo” -> orchestral music
- “Footsteps on gravel with birds chirping” -> detailed soundscape
- “Electric guitar riff with heavy distortion” -> rock music
The dual encoder architecture means:
- CLAP encoder understands audio concepts (instrument sounds, effects)
- T5/GPT-2 encoder understands language (descriptions, context)
- Combined, they produce audio that matches both sound and meaning
How It Works
AudioLDM 2 is an improved version of AudioLDM with significant architectural enhancements for better text-to-audio and text-to-music generation. Key improvements include:
- Dual Text Encoders: Combines CLAP (audio-text) and T5/GPT-2 (language) embeddings
- Larger Architecture: 384 base channels vs 256 in AudioLDM 1
- Higher Resolution: 128 mel channels vs 64 for better audio quality
- Improved Music Generation: Better temporal coherence and musical structure
- Longer Duration Support: Up to 30 seconds of audio generation
Technical specifications:
- Sample rate: 16 kHz (speech/effects) or 48 kHz (high-quality music)
- Latent channels: 8
- Mel channels: 128 (double AudioLDM 1)
- Base channels: 384 (1.5x AudioLDM 1)
- Context dimension: 1024 (combined encoder output)
- Duration: Up to 30 seconds
- Guidance scale: 3.0-6.0 typical
Example
// Create an AudioLDM 2 model
var audioLDM2 = new AudioLDM2Model<float>();
// Generate high-quality music
var music = audioLDM2.GenerateMusic(
prompt: "Cinematic orchestral music with dramatic strings and timpani",
durationSeconds: 20.0,
numInferenceSteps: 200,
guidanceScale: 4.5);
// Generate complex sound effects
var soundscape = audioLDM2.GenerateAudio(
prompt: "A busy city street with traffic, horns, and people talking",
durationSeconds: 15.0,
numInferenceSteps: 150,
guidanceScale: 4.0);
Constructors
| Constructor | Summary |
|---|---|
AudioLDM2Model(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> unet, AudioVAE<T> audioVAE, IConditioningModule<T> clapConditioner, IConditioningModule<T> languageConditioner, AudioLDM2Variant variant, int sampleRate, double defaultDurationSeconds, int? seed) |
Initializes a new AudioLDM 2 model with full customization support. |
Properties
| Property | Summary |
|---|---|
AudioLDM2Variant Variant { get; } |
Gets the model variant. |
AudioVAE<T> AudioVAE { get; } |
Gets the AudioVAE for direct access. |
IConditioningModule<T> Conditioner { get; } |
|
IConditioningModule<T> LanguageConditioner { get; } |
Gets the secondary language conditioning module. |
INoisePredictor<T> NoisePredictor { get; } |
|
IVAEModel<T> VAE { get; } |
|
bool SupportsAudioToAudio { get; } |
|
bool SupportsTextToAudio { get; } |
|
bool SupportsTextToMusic { get; } |
|
bool SupportsTextToSpeech { get; } |
|
int LatentChannels { get; } |
Methods
| Method | Summary |
|---|---|
List<Tensor<T>> GenerateVariations(Tensor<T> inputAudio, int numVariations, double variationStrength, int? seed) |
Generates audio variations with enhanced diversity. |
List<Tensor<T>> InterpolateAudio(Tensor<T> audio1, Tensor<T> audio2, int numSteps) |
Interpolates between two audio samples in latent space. |
Tensor<T> GenerateAudio(string prompt, string negativePrompt, double? durationSeconds, int numInferenceSteps, double guidanceScale, int? seed) |
Generates audio from a text prompt using dual encoders. |
Tensor<T> GenerateMusic(string prompt, string negativePrompt, double? durationSeconds, int numInferenceSteps, double guidanceScale, int? seed) |
Generates music from a text prompt with enhanced musical understanding. |
Tensor<T> TransformAudio(Tensor<T> inputAudio, string prompt, string negativePrompt, double strength, int numInferenceSteps, double guidanceScale, int? seed) |
Transforms audio based on a text prompt (audio-to-audio). |
Fields
| Field | Summary |
|---|---|
double AUDIOLDM2_MAX_DURATION |
Maximum supported duration in seconds. |
int AUDIOLDM2_BASE_CHANNELS |
AudioLDM 2 U-Net base channels (larger than AudioLDM 1). |
int AUDIOLDM2_CONTEXT_DIM |
Combined context dimension from dual encoders. |
int AUDIOLDM2_LATENT_CHANNELS |
AudioLDM 2 latent space channels. |
int AUDIOLDM2_MEL_CHANNELS |
AudioLDM 2 mel spectrogram channels (increased from 64 to 128). |
int AUDIOLDM2_SAMPLE_RATE |
AudioLDM 2 default sample rate for high-quality audio. |