AudioGenModel<T>
Models & Types · AiDotNet.Audio.AudioGen
AudioGen model for generating audio from text descriptions using neural audio codecs.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → AudioGenModel<T>
Implements: IAudioGenerator<T>, IGeneratedParameterRegistrar<T>
For Beginners
AudioGen is fundamentally different from Text-to-Speech (TTS):
TTS vs AudioGen:
- TTS: Converts specific words to speech (“Hello world” -> spoken words “Hello world”)
- AudioGen: Creates sounds matching a description (“dog barking” -> actual bark sound)
How it works:
- Your text prompt (“a cat meowing softly”) is encoded into a numerical representation
- A language model generates a sequence of “audio tokens” (like words, but for sound)
- The EnCodec decoder converts these tokens back into actual audio waveforms
Why discrete audio codes?
- Raw audio has too many samples (32,000 per second!)
- EnCodec compresses audio to ~50 tokens per second
- This makes the language model’s job much easier
Common use cases:
- Sound effect generation for games/films
- Creating ambient soundscapes
- Generating audio for multimedia content
- Rapid prototyping of audio concepts
Limitations:
- Cannot generate intelligible speech (use TTS for that)
- Quality depends on training data
- May struggle with very specific or unusual sounds
How It Works
AudioGen uses a language model approach to generate audio from text prompts. The architecture consists of three main components:
- Text Encoder: Converts text prompts to embeddings (typically T5-based)
- Audio Language Model: Generates discrete audio codes autoregressively
- Audio Decoder (EnCodec): Converts audio codes back to waveforms
Reference: “AudioGen: Textually Guided Audio Generation” by Kreuk et al., 2022
Example
// Create an AudioGen model for generating audio from text descriptions
var architecture = new NeuralNetworkArchitecture<float>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.Generative,
inputSize: 512,
outputSize: 32000);
var model = new AudioGenModel<float>(
architecture: architecture,
textEncoderPath: "text_encoder.onnx",
languageModelPath: "language_model.onnx",
audioDecoderPath: "audio_decoder.onnx",
tokenizer: CharacterTokenizer.CreateAscii());
// Generate audio from a text prompt
Tensor<float> audio = model.GenerateAudio("a dog barking loudly");
Constructors
| Constructor | Summary |
|---|---|
AudioGenModel(NeuralNetworkArchitecture<T> architecture, AudioGenOptions options, int? seed, ITokenizer tokenizer, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
Creates an AudioGen network using native library layers for training from scratch. |
AudioGenModel(NeuralNetworkArchitecture<T> architecture, string textEncoderPath, string languageModelPath, string audioDecoderPath, ITokenizer tokenizer, AudioGenOptions options, int? seed, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) |
Creates an AudioGen network using pretrained ONNX models. |
Properties
| Property | Summary |
|---|---|
AudioGenModelSize ModelSize { get; } |
Gets the model size variant. |
bool IsReady { get; } |
Gets whether the model is ready for inference. |
bool SupportsAudioContinuation { get; } |
Gets whether this model supports audio continuation. |
bool SupportsAudioInpainting { get; } |
Gets whether this model supports audio inpainting. |
bool SupportsTextToAudio { get; } |
Gets whether this model supports text-to-audio generation. |
bool SupportsTextToMusic { get; } |
Gets whether this model supports text-to-music generation. |
double MaxDurationSeconds { get; } |
Gets the maximum duration of audio that can be generated in seconds. |
Methods
| Method | Summary |
|---|---|
AudioGenerationOptions<T> GetDefaultOptions() |
Gets default generation options. |
ModelMetadata<T> GetModelMetadata() |
Gets metadata about the model. |
Task<Tensor<T>> GenerateAudioAsync(string prompt, string negativePrompt, double durationSeconds, int numInferenceSteps, double guidanceScale, int? seed, CancellationToken cancellationToken) |
Generates audio from a text description asynchronously. |
Tensor<T> ContinueAudio(Tensor<T> inputAudio, string prompt, double extensionSeconds, int numInferenceSteps, int? seed) |
Continues existing audio to extend it naturally. |
Tensor<T> GenerateAudio(string prompt, string negativePrompt, double durationSeconds, int numInferenceSteps, double guidanceScale, int? seed) |
Generates audio from a text description. |
Tensor<T> GenerateMusic(string prompt, string negativePrompt, double durationSeconds, int numInferenceSteps, double guidanceScale, int? seed) |
Generates music from a text description. |
Tensor<T> InpaintAudio(Tensor<T> audio, Tensor<T> mask, string prompt, int numInferenceSteps, int? seed) |
Fills in missing or masked sections of audio. |