AudioDiffusionModelBase<T>
Base Classes · AiDotNet.Diffusion
Base class for audio diffusion models that generate sound and music.
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → AudioDiffusionModelBase<T>
Implements: IAudioDiffusionModel<T>
Known derived types: AudioLDM2Model<T>, AudioLDMModel<T>, JEN1Model<T>, StableAudioModel<T>, UdioModel<T>
For Beginners
This is the foundation for audio generation models like AudioLDM. It extends latent diffusion to work with audio by converting sound to spectrograms (visual representations of sound) and back.
How It Works
This abstract base class provides common functionality for all audio diffusion models, including text-to-audio generation, text-to-music, text-to-speech, and audio transformation.
How audio diffusion works:
- Audio is converted to a mel spectrogram (frequency vs time image)
- The spectrogram is encoded to latent space (like images)
- Diffusion denoising happens in latent space
- The result is decoded to a spectrogram
- A vocoder converts the spectrogram back to audio
Properties
| Property | Summary |
|---|---|
bool SupportsAudioToAudio { get; } |
|
bool SupportsTextToAudio { get; } |
|
bool SupportsTextToMusic { get; } |
|
bool SupportsTextToSpeech { get; } |
|
double DefaultDurationSeconds { get; } |
|
double MaxFrequency { get; set; } |
Gets the maximum frequency for mel filterbank. |
double MinFrequency { get; set; } |
Gets the minimum frequency for mel filterbank. |
int FFTSize { get; set; } |
Gets the FFT window size. |
int HopLength { get; set; } |
Gets the hop length for spectrogram computation. |
int MelChannels { get; } |
|
int SampleRate { get; } |
Methods
| Method | Summary |
|---|---|
Tensor<T> AudioToAudio(Tensor<T> inputAudio, string prompt, string negativePrompt, double strength, int numInferenceSteps, double guidanceScale, int? seed) |
|
Tensor<T> ContinueAudio(Tensor<T> inputAudio, string prompt, double extensionSeconds, int numInferenceSteps, int? seed) |
|
Tensor<T> ExtractSpeakerEmbedding(Tensor<T> referenceAudio) |
|
Tensor<T> GenerateFromText(string prompt, string negativePrompt, double? durationSeconds, int numInferenceSteps, double guidanceScale, int? seed) |
|
Tensor<T> GenerateMusic(string prompt, string negativePrompt, double? durationSeconds, int numInferenceSteps, double guidanceScale, int? seed) |
|
Tensor<T> MelSpectrogramToWaveform(Tensor<T> melSpectrogram) |
Copies a mel spectrogram to a specific batch position. |
Tensor<T> TextToSpeech(string text, Tensor<T> speakerEmbedding, double speakingRate, int numInferenceSteps, int? seed) |
|
Tensor<T> WaveformToMelSpectrogram(Tensor<T> waveform) |