AudioDiffusionModelBase<T>

Base Classes · AiDotNet.Diffusion

Base class for audio diffusion models that generate sound and music.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → AudioDiffusionModelBase<T>

Implements: IAudioDiffusionModel<T>

Known derived types: AudioLDM2Model<T>, AudioLDMModel<T>, JEN1Model<T>, StableAudioModel<T>, UdioModel<T>

For Beginners

This is the foundation for audio generation models like AudioLDM. It extends latent diffusion to work with audio by converting sound to spectrograms (visual representations of sound) and back.

How It Works

This abstract base class provides common functionality for all audio diffusion models, including text-to-audio generation, text-to-music, text-to-speech, and audio transformation.

How audio diffusion works:

  1. Audio is converted to a mel spectrogram (frequency vs time image)
  2. The spectrogram is encoded to latent space (like images)
  3. Diffusion denoising happens in latent space
  4. The result is decoded to a spectrogram
  5. A vocoder converts the spectrogram back to audio

Properties

Property Summary
bool SupportsAudioToAudio { get; }
bool SupportsTextToAudio { get; }
bool SupportsTextToMusic { get; }
bool SupportsTextToSpeech { get; }
double DefaultDurationSeconds { get; }
double MaxFrequency { get; set; } Gets the maximum frequency for mel filterbank.
double MinFrequency { get; set; } Gets the minimum frequency for mel filterbank.
int FFTSize { get; set; } Gets the FFT window size.
int HopLength { get; set; } Gets the hop length for spectrogram computation.
int MelChannels { get; }
int SampleRate { get; }

Methods

Method Summary
Tensor<T> AudioToAudio(Tensor<T> inputAudio, string prompt, string negativePrompt, double strength, int numInferenceSteps, double guidanceScale, int? seed)
Tensor<T> ContinueAudio(Tensor<T> inputAudio, string prompt, double extensionSeconds, int numInferenceSteps, int? seed)
Tensor<T> ExtractSpeakerEmbedding(Tensor<T> referenceAudio)
Tensor<T> GenerateFromText(string prompt, string negativePrompt, double? durationSeconds, int numInferenceSteps, double guidanceScale, int? seed)
Tensor<T> GenerateMusic(string prompt, string negativePrompt, double? durationSeconds, int numInferenceSteps, double guidanceScale, int? seed)
Tensor<T> MelSpectrogramToWaveform(Tensor<T> melSpectrogram) Copies a mel spectrogram to a specific batch position.
Tensor<T> TextToSpeech(string text, Tensor<T> speakerEmbedding, double speakingRate, int numInferenceSteps, int? seed)
Tensor<T> WaveformToMelSpectrogram(Tensor<T> waveform)