AudioLDMModel<T>
Models & Types · AiDotNet.Diffusion.Audio
Audio Latent Diffusion Model (AudioLDM) for text-to-audio generation.
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → AudioDiffusionModelBase<T> → AudioLDMModel<T>
For Beginners
AudioLDM lets you create sounds and music from text descriptions:
Example prompts:
- “A dog barking in a park” -> generates dog barking sounds
- “Rain falling on a window” -> generates rain sounds
- “Jazz piano playing softly” -> generates jazz piano music
How it works:
- Text -> CLAP encoder -> text embedding (understands audio concepts)
- Text embedding guides diffusion in latent space
- Latent -> AudioVAE decoder -> mel spectrogram
- Mel spectrogram -> Vocoder -> audio waveform
Key features:
- Text-to-audio: Generate sounds from descriptions
- Audio-to-audio: Transform sounds while preserving some characteristics
- Variable duration: Generate audio of different lengths
- Classifier-free guidance: Control how closely to follow the prompt
How It Works
AudioLDM is a latent diffusion model specifically designed for audio generation. It works by generating mel spectrograms in latent space and then converting them to audio using a vocoder (like HiFi-GAN).
Technical specifications:
- Sample rate: 16 kHz (standard for speech/effects) or 48 kHz (music)
- Latent channels: 8
- Mel channels: 64 (AudioLDM) or 128 (AudioLDM 2)
- Duration: Typically 10 seconds, but configurable
- Guidance scale: 2.5-5.0 typical
Example
// Create an AudioLDM model
var audioLDM = new AudioLDMModel<float>();
// Generate sound effects
var dogBark = audioLDM.GenerateFromText(
prompt: "A dog barking excitedly",
durationSeconds: 5.0,
numInferenceSteps: 100,
guidanceScale: 3.5);
// Generate music
var music = audioLDM.GenerateMusic(
prompt: "Soft jazz piano with light drums",
durationSeconds: 10.0,
numInferenceSteps: 200,
guidanceScale: 4.0);
// Save as audio file
// dogBark is the result tensor — write it out with your own encoder.
Constructors
| Constructor | Summary |
|---|---|
AudioLDMModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> unet, AudioVAE<T> audioVAE, IConditioningModule<T> conditioner, int sampleRate, double defaultDurationSeconds, int melChannels, bool isVersion2, int? seed) |
Initializes a new AudioLDM model with full customization support. |
Properties
| Property | Summary |
|---|---|
AudioVAE<T> AudioVAE { get; } |
Gets the AudioVAE used for encoding/decoding. |
IConditioningModule<T> Conditioner { get; } |
|
INoisePredictor<T> NoisePredictor { get; } |
|
IVAEModel<T> VAE { get; } |
|
bool IsVersion2 { get; } |
Gets whether this is AudioLDM version 2. |
bool SupportsAudioToAudio { get; } |
|
bool SupportsTextToAudio { get; } |
|
bool SupportsTextToMusic { get; } |
|
bool SupportsTextToSpeech { get; } |
|
int LatentChannels { get; } |
Methods
| Method | Summary |
|---|---|
List<Tensor<T>> GenerateVariations(Tensor<T> inputAudio, int numVariations, double variationStrength, int? seed) |
Generates audio variations from an input audio. |
Tensor<T> GenerateAudio(string prompt, string negativePrompt, double? durationSeconds, int numInferenceSteps, double guidanceScale, int? seed) |
Generates audio from a text prompt. |
Tensor<T> GenerateMusic(string prompt, string negativePrompt, double? durationSeconds, int numInferenceSteps, double guidanceScale, int? seed) |
Generates music from a text prompt. |
Tensor<T> TransformAudio(Tensor<T> inputAudio, string prompt, string negativePrompt, double strength, int numInferenceSteps, double guidanceScale, int? seed) |
Transforms audio based on a text prompt (audio-to-audio). |
Fields
| Field | Summary |
|---|---|
int AUDIOLDM_LATENT_CHANNELS |
Standard AudioLDM latent channels. |
int AUDIOLDM_MEL_CHANNELS |
Standard AudioLDM mel channels. |
int AUDIOLDM_SAMPLE_RATE |
Standard AudioLDM sample rate. |