IAudioGenerator<T>

Interfaces · AiDotNet.Interfaces

Interface for audio generation models that create audio from text descriptions or other conditions.

Implements: IFeatureAware, IFullModel<T, Tensor<T>, Tensor<T>>, IGradientComputable<T, Tensor<T>, Tensor<T>>

Known implementations: ACEStep<T>, AudioGenModel<T>, AudioLM<T>, MusicGenModel<T>, StableAudioModel<T>, YuE<T>

For Beginners

Audio generation is like having an artist who can create any sound you describe.

How audio generation works:

  1. You provide a description (“A dog barking in a park”)
  2. The model generates audio features that match the description
  3. The features are converted to playable audio

Types of audio generation:

  • Text-to-Audio: “Thunder during a storm” creates thunder sounds
  • Text-to-Music: “Upbeat jazz piano” creates music
  • Audio Inpainting: Fill in missing parts of audio
  • Audio Continuation: Extend existing audio naturally

Common use cases:

  • Video game sound effects
  • Film and media production
  • Music composition assistance
  • Podcast and content creation

How It Works

Audio generation models create sounds, music, and audio effects from various inputs. Unlike TTS which focuses on speech, audio generators can produce any type of sound including music, environmental sounds, and sound effects.

This interface extends IFullModel for Tensor-based audio processing.

Properties

Property Summary
bool IsOnnxMode { get; } Gets whether this model is running in ONNX inference mode.
bool SupportsAudioContinuation { get; } Gets whether this model supports audio continuation.
bool SupportsAudioInpainting { get; } Gets whether this model supports audio inpainting.
bool SupportsTextToAudio { get; } Gets whether this model supports text-to-audio generation.
bool SupportsTextToMusic { get; } Gets whether this model supports text-to-music generation.
double MaxDurationSeconds { get; } Gets the maximum duration of audio that can be generated in seconds.
int SampleRate { get; } Gets the sample rate of generated audio.

Methods

Method Summary
AudioGenerationOptions<T> GetDefaultOptions() Gets generation options for advanced control.
Task<Tensor<T>> GenerateAudioAsync(string prompt, string negativePrompt, double durationSeconds, int numInferenceSteps, double guidanceScale, int? seed, CancellationToken cancellationToken) Generates audio from a text description asynchronously.
Tensor<T> ContinueAudio(Tensor<T> inputAudio, string prompt, double extensionSeconds, int numInferenceSteps, int? seed) Continues existing audio to extend it naturally.
Tensor<T> GenerateAudio(string prompt, string negativePrompt, double durationSeconds, int numInferenceSteps, double guidanceScale, int? seed) Generates audio from a text description.
Tensor<T> GenerateMusic(string prompt, string negativePrompt, double durationSeconds, int numInferenceSteps, double guidanceScale, int? seed) Generates music from a text description.
Tensor<T> InpaintAudio(Tensor<T> audio, Tensor<T> mask, string prompt, int numInferenceSteps, int? seed) Fills in missing or masked sections of audio.