AudioGenModel<T>

Models & Types · AiDotNet.Audio.AudioGen

AudioGen model for generating audio from text descriptions using neural audio codecs.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → AudioGenModel<T>

Implements: IAudioGenerator<T>, IGeneratedParameterRegistrar<T>

For Beginners

AudioGen is fundamentally different from Text-to-Speech (TTS):

TTS vs AudioGen:

  • TTS: Converts specific words to speech (“Hello world” -> spoken words “Hello world”)
  • AudioGen: Creates sounds matching a description (“dog barking” -> actual bark sound)

How it works:

  1. Your text prompt (“a cat meowing softly”) is encoded into a numerical representation
  2. A language model generates a sequence of “audio tokens” (like words, but for sound)
  3. The EnCodec decoder converts these tokens back into actual audio waveforms

Why discrete audio codes?

  • Raw audio has too many samples (32,000 per second!)
  • EnCodec compresses audio to ~50 tokens per second
  • This makes the language model’s job much easier

Common use cases:

  • Sound effect generation for games/films
  • Creating ambient soundscapes
  • Generating audio for multimedia content
  • Rapid prototyping of audio concepts

Limitations:

  • Cannot generate intelligible speech (use TTS for that)
  • Quality depends on training data
  • May struggle with very specific or unusual sounds

How It Works

AudioGen uses a language model approach to generate audio from text prompts. The architecture consists of three main components:

  • Text Encoder: Converts text prompts to embeddings (typically T5-based)
  • Audio Language Model: Generates discrete audio codes autoregressively
  • Audio Decoder (EnCodec): Converts audio codes back to waveforms

Reference: “AudioGen: Textually Guided Audio Generation” by Kreuk et al., 2022

Example

// Create an AudioGen model for generating audio from text descriptions
var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.Generative,
    inputSize: 512,
    outputSize: 32000);
            
var model = new AudioGenModel<float>(
    architecture: architecture,
    textEncoderPath: "text_encoder.onnx",
    languageModelPath: "language_model.onnx",
    audioDecoderPath: "audio_decoder.onnx",
    tokenizer: CharacterTokenizer.CreateAscii());
            
// Generate audio from a text prompt
Tensor<float> audio = model.GenerateAudio("a dog barking loudly");

Constructors

Constructor Summary
AudioGenModel(NeuralNetworkArchitecture<T> architecture, AudioGenOptions options, int? seed, ITokenizer tokenizer, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates an AudioGen network using native library layers for training from scratch.
AudioGenModel(NeuralNetworkArchitecture<T> architecture, string textEncoderPath, string languageModelPath, string audioDecoderPath, ITokenizer tokenizer, AudioGenOptions options, int? seed, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates an AudioGen network using pretrained ONNX models.

Properties

Property Summary
AudioGenModelSize ModelSize { get; } Gets the model size variant.
bool IsReady { get; } Gets whether the model is ready for inference.
bool SupportsAudioContinuation { get; } Gets whether this model supports audio continuation.
bool SupportsAudioInpainting { get; } Gets whether this model supports audio inpainting.
bool SupportsTextToAudio { get; } Gets whether this model supports text-to-audio generation.
bool SupportsTextToMusic { get; } Gets whether this model supports text-to-music generation.
double MaxDurationSeconds { get; } Gets the maximum duration of audio that can be generated in seconds.

Methods

Method Summary
AudioGenerationOptions<T> GetDefaultOptions() Gets default generation options.
ModelMetadata<T> GetModelMetadata() Gets metadata about the model.
Task<Tensor<T>> GenerateAudioAsync(string prompt, string negativePrompt, double durationSeconds, int numInferenceSteps, double guidanceScale, int? seed, CancellationToken cancellationToken) Generates audio from a text description asynchronously.
Tensor<T> ContinueAudio(Tensor<T> inputAudio, string prompt, double extensionSeconds, int numInferenceSteps, int? seed) Continues existing audio to extend it naturally.
Tensor<T> GenerateAudio(string prompt, string negativePrompt, double durationSeconds, int numInferenceSteps, double guidanceScale, int? seed) Generates audio from a text description.
Tensor<T> GenerateMusic(string prompt, string negativePrompt, double durationSeconds, int numInferenceSteps, double guidanceScale, int? seed) Generates music from a text description.
Tensor<T> InpaintAudio(Tensor<T> audio, Tensor<T> mask, string prompt, int numInferenceSteps, int? seed) Fills in missing or masked sections of audio.