RiffusionModel<T>

Models & Types · AiDotNet.Diffusion.Audio

Riffusion model for music generation via spectrogram diffusion.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → RiffusionModel<T>

For Beginners

Riffusion creates music by first generating a “picture” of the sound (spectrogram), then converting that picture back into actual audio.

How it works:

  1. You describe the music you want: “jazz piano solo”
  2. Riffusion generates a spectrogram (visual representation of sound)
  3. The spectrogram is converted to playable audio

Key features:

  • Text-to-music generation
  • Style interpolation (blend two music styles)
  • Real-time streaming generation
  • Works with any Stable Diffusion checkpoint

What makes it unique:

  • Treats audio generation as an image generation problem
  • Can leverage all SD techniques: ControlNet, img2img, etc.
  • Fast inference compared to autoregressive music models

How It Works

Riffusion generates music by treating audio spectrograms as images and using Stable Diffusion to generate them. The resulting spectrograms are then converted back to audio using the Griffin-Lim algorithm or neural vocoders.

Technical details:

  • Uses mel-spectrograms with specific parameters
  • Typically 512x512 spectrogram images
  • Griffin-Lim or neural vocoder for audio reconstruction
  • Supports seed-based interpolation for smooth transitions
  • Compatible with LoRA adapters for style transfer

Reference: Based on Riffusion project (riffusion.com)

Example

// Create a Riffusion model
var riffusion = new RiffusionModel<float>();
            
// Generate music from text
var spectrogram = riffusion.GenerateSpectrogram(
    prompt: "jazz piano solo, smooth and relaxing",
    durationSeconds: 5.0);
            
// Convert to audio
var audio = riffusion.SpectrogramToAudio(spectrogram);
            
// Interpolate between two styles
var interpolated = riffusion.InterpolateStyles(
    promptA: "upbeat electronic dance music",
    promptB: "calm ambient soundscape",
    alpha: 0.5);

Constructors

Constructor Summary
RiffusionModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> unet, StandardVAE<T> vae, IConditioningModule<T> conditioner, SpectrogramConfig spectrogramConfig, int? seed) Initializes a new instance of RiffusionModel with full customization support.

Properties

Property Summary
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> VAE { get; }
SpectrogramConfig SpectrogramConfiguration { get; } Gets the spectrogram configuration.
int LatentChannels { get; }

Methods

Method Summary
Tensor<T> GenerateAudio(string prompt, string negativePrompt, double durationSeconds, int numInferenceSteps, double? guidanceScale, int? seed) Generates audio directly from a text prompt.
Tensor<T> GenerateSpectrogram(string prompt, string negativePrompt, double durationSeconds, int numInferenceSteps, double? guidanceScale, int? seed) Generates a spectrogram from a text prompt.
Tensor<T> InterpolateStyles(string promptA, string promptB, double alpha, double durationSeconds, int numInferenceSteps, double? guidanceScale, int? seed) Interpolates between two music styles.
Tensor<T> SpectrogramToAudio(Tensor<T> spectrogram) Converts a spectrogram to audio waveform.