StableAudioModel<T>

Models & Types · AiDotNet.Diffusion.Audio

Stable Audio Open model — DiT-based latent diffusion for long-form audio generation.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → AudioDiffusionModelBase<T> → StableAudioModel<T>

For Beginners

Stable Audio creates music and sound effects from text:

How Stable Audio works:

  1. Text is encoded using a CLAP text encoder for audio-aware embeddings
  2. A DiT (Diffusion Transformer) denoises latents conditioned on text and timing
  3. Timing embeddings (start_s, total_s) control where and how long to generate
  4. An autoencoder decodes the latents back to 44.1 kHz stereo audio

Key characteristics:

  • DiT backbone with timing conditioning (start_s, total_s)
  • 44.1 kHz sample rate for high-fidelity audio
  • Autoencoder-based latent space
  • Up to 47 seconds of stereo audio
  • Supports both music and sound effect generation

Advantages:

  • High audio quality at 44.1 kHz
  • Variable-length generation via timing conditioning
  • Strong prompt adherence through CLAP embeddings
  • Open-source weights and architecture

Limitations:

  • Maximum duration of 47 seconds
  • No text-to-speech capability
  • Requires significant compute for real-time generation

Reference: Evans et al., “Stable Audio Open”, Stability AI, 2024

How It Works

Stable Audio uses a DiT operating on audio latents with timing conditioning for variable-length, high-quality audio generation at 44.1 kHz.

Example

var originalAudio = Tensor<float>.CreateRandom(1, 16000);
// Create a Stable Audio model with default settings
var stableAudio = new StableAudioModel<float>();
            
// Generate ambient music from a text prompt
var music = stableAudio.GenerateFromText(
    prompt: "Ambient electronic music with soft pads and gentle arpeggios",
    durationSeconds: 30.0,
    numInferenceSteps: 100,
    guidanceScale: 3.5,
    seed: 42);
            
// Transform existing audio with a new style
var transformed = stableAudio.AudioToAudio(
    inputAudio: originalAudio,
    prompt: "Lo-fi version with vinyl crackle",
    strength: 0.6);

Constructors

Constructor Summary
StableAudioModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, DiTNoisePredictor<T> dit, AudioVAE<T> audioVAE, IConditioningModule<T> conditioner) Initializes a new instance of StableAudioModel with full customization support.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> VAE { get; }
bool SupportsAudioToAudio { get; }
bool SupportsTextToAudio { get; }
bool SupportsTextToMusic { get; }
bool SupportsTextToSpeech { get; }
int LatentChannels { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()