StableAudioModel<T>

Models & Types · AiDotNet.Audio.StableAudio

Stable Audio model for generating high-quality audio from text descriptions.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → StableAudioModel<T>

Implements: IAudioGenerator<T>

For Beginners

Stable Audio creates professional-quality audio:

How it works:

  1. You describe the audio you want (“upbeat electronic track”)
  2. T5 encodes your text into embeddings
  3. Duration and timing are encoded as conditioning
  4. DiT diffusion generates latent audio representations
  5. VAE decoder converts latents to 44.1kHz stereo audio

Key features:

  • CD-quality 44.1kHz stereo output
  • Variable-length generation (up to 3 minutes)
  • Music and sound effects generation
  • Timing-aware conditioning

Usage:

How It Works

Stable Audio is Stability AI’s state-of-the-art audio generation model that uses latent diffusion with a Diffusion Transformer (DiT) architecture for high-quality music and sound effects generation.

Architecture components:

  • T5 Text Encoder: Encodes text prompts into conditioning embeddings
  • VAE: Compresses audio to/from latent space (44.1kHz to 21.5Hz latent)
  • DiT (Diffusion Transformer): Predicts noise using transformer blocks
  • Timing Conditioning: Encodes duration and timing information

Reference: “Stable Audio: Fast Timing-Conditioned Latent Audio Diffusion” by Evans et al., 2024

Constructors

Constructor Summary
StableAudioModel(NeuralNetworkArchitecture<T> architecture, StableAudioOptions options, ITokenizer tokenizer, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a Stable Audio model using native layers for training from scratch.
StableAudioModel(NeuralNetworkArchitecture<T> architecture, string textEncoderPath, string vaePath, string ditPath, ITokenizer tokenizer, StableAudioOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a Stable Audio model using pretrained ONNX models for inference.

Properties

Property Summary
bool SupportsAudioContinuation { get; } Gets whether this model supports audio continuation.
bool SupportsAudioInpainting { get; } Gets whether this model supports audio inpainting.
bool SupportsTextToAudio { get; } Gets whether this model supports text-to-audio generation.
bool SupportsTextToMusic { get; } Gets whether this model supports text-to-music generation.
double MaxDurationSeconds { get; } Gets the maximum duration of audio that can be generated.
int SampleRate { get; } Gets the sample rate of generated audio.

Methods

Method Summary
AudioGenerationOptions<T> GetDefaultOptions() Gets default generation options.
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
ModelMetadata<T> GetModelMetadata() Gets model metadata.
Task<Tensor<T>> GenerateAudioAsync(string prompt, string negativePrompt, double durationSeconds, int numInferenceSteps, double guidanceScale, int? seed, CancellationToken cancellationToken) Generates audio asynchronously.
Tensor<T> ContinueAudio(Tensor<T> inputAudio, string prompt, double extensionSeconds, int numInferenceSteps, int? seed) Continues existing audio by extending it.
Tensor<T> GenerateAudio(string prompt, string negativePrompt, double durationSeconds, int numInferenceSteps, double guidanceScale, int? seed) Generates audio from a text description.
Tensor<T> GenerateMusic(string prompt, string negativePrompt, double durationSeconds, int numInferenceSteps, double guidanceScale, int? seed) Generates music from a text description.
Tensor<T> InpaintAudio(Tensor<T> audio, Tensor<T> mask, string prompt, int numInferenceSteps, int? seed) Fills in missing or masked sections of audio.