IAudioDiffusionModel<T>
Interfaces · AiDotNet.Interfaces
Interface for audio diffusion models that generate sound and music.
Implements: IDiffusionModel<T>
Known implementations: AudioDiffusionModelBase<T>
For Beginners
Audio diffusion models work similarly to image diffusion, but instead of generating pictures, they create sounds.
How audio diffusion works:
- Audio is converted to a spectrogram (visual representation of sound)
- Diffusion happens on this spectrogram (just like image diffusion)
- The spectrogram is converted back to audio
Types of audio generation:
- Text-to-Audio: “A dog barking in a park” → audio clip
- Text-to-Music: “Upbeat jazz piano” → music track
- Text-to-Speech: Text → spoken voice
- Audio-to-Audio: Transform existing audio (voice conversion, style transfer)
Key challenges:
- Temporal coherence (sounds must flow naturally)
- Frequency relationships (harmonics, rhythm)
- Long-range dependencies (verse-chorus structure in music)
How It Works
Audio diffusion models apply diffusion processes to generate audio content, including music, speech, sound effects, and more. They typically operate on audio spectrograms or mel-spectrograms in latent space.
This interface extends IDiffusionModel with audio-specific operations.
Properties
| Property | Summary |
|---|---|
bool SupportsAudioToAudio { get; } |
Gets whether this model supports audio-to-audio transformation. |
bool SupportsTextToAudio { get; } |
Gets whether this model supports text-to-audio generation. |
bool SupportsTextToMusic { get; } |
Gets whether this model supports text-to-music generation. |
bool SupportsTextToSpeech { get; } |
Gets whether this model supports text-to-speech generation. |
double DefaultDurationSeconds { get; } |
Gets the default duration of generated audio in seconds. |
int MelChannels { get; } |
Gets the number of mel spectrogram channels used. |
int SampleRate { get; } |
Gets the sample rate of generated audio. |
Methods
| Method | Summary |
|---|---|
Tensor<T> AudioToAudio(Tensor<T> inputAudio, string prompt, string negativePrompt, double strength, int numInferenceSteps, double guidanceScale, int? seed) |
Transforms existing audio based on a text prompt. |
Tensor<T> ContinueAudio(Tensor<T> inputAudio, string prompt, double extensionSeconds, int numInferenceSteps, int? seed) |
Continues/extends audio from a given clip. |
Tensor<T> ExtractSpeakerEmbedding(Tensor<T> referenceAudio) |
Gets speaker embeddings from a reference audio clip (for voice cloning). |
Tensor<T> GenerateFromText(string prompt, string negativePrompt, double? durationSeconds, int numInferenceSteps, double guidanceScale, int? seed) |
Generates audio from a text description. |
Tensor<T> GenerateMusic(string prompt, string negativePrompt, double? durationSeconds, int numInferenceSteps, double guidanceScale, int? seed) |
Generates music from a text description. |
Tensor<T> MelSpectrogramToWaveform(Tensor<T> melSpectrogram) |
Converts mel spectrogram back to audio waveform. |
Tensor<T> TextToSpeech(string text, Tensor<T> speakerEmbedding, double speakingRate, int numInferenceSteps, int? seed) |
Synthesizes speech from text (text-to-speech). |
Tensor<T> WaveformToMelSpectrogram(Tensor<T> waveform) |
Converts audio waveform to mel spectrogram. |