RiffusionModel<T>
Models & Types · AiDotNet.Diffusion.Audio
Riffusion model for music generation via spectrogram diffusion.
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → RiffusionModel<T>
For Beginners
Riffusion creates music by first generating a “picture” of the sound (spectrogram), then converting that picture back into actual audio.
How it works:
- You describe the music you want: “jazz piano solo”
- Riffusion generates a spectrogram (visual representation of sound)
- The spectrogram is converted to playable audio
Key features:
- Text-to-music generation
- Style interpolation (blend two music styles)
- Real-time streaming generation
- Works with any Stable Diffusion checkpoint
What makes it unique:
- Treats audio generation as an image generation problem
- Can leverage all SD techniques: ControlNet, img2img, etc.
- Fast inference compared to autoregressive music models
How It Works
Riffusion generates music by treating audio spectrograms as images and using Stable Diffusion to generate them. The resulting spectrograms are then converted back to audio using the Griffin-Lim algorithm or neural vocoders.
Technical details:
- Uses mel-spectrograms with specific parameters
- Typically 512x512 spectrogram images
- Griffin-Lim or neural vocoder for audio reconstruction
- Supports seed-based interpolation for smooth transitions
- Compatible with LoRA adapters for style transfer
Reference: Based on Riffusion project (riffusion.com)
Example
// Create a Riffusion model
var riffusion = new RiffusionModel<float>();
// Generate music from text
var spectrogram = riffusion.GenerateSpectrogram(
prompt: "jazz piano solo, smooth and relaxing",
durationSeconds: 5.0);
// Convert to audio
var audio = riffusion.SpectrogramToAudio(spectrogram);
// Interpolate between two styles
var interpolated = riffusion.InterpolateStyles(
promptA: "upbeat electronic dance music",
promptB: "calm ambient soundscape",
alpha: 0.5);
Constructors
| Constructor | Summary |
|---|---|
RiffusionModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> unet, StandardVAE<T> vae, IConditioningModule<T> conditioner, SpectrogramConfig spectrogramConfig, int? seed) |
Initializes a new instance of RiffusionModel with full customization support. |
Properties
| Property | Summary |
|---|---|
IConditioningModule<T> Conditioner { get; } |
|
INoisePredictor<T> NoisePredictor { get; } |
|
IVAEModel<T> VAE { get; } |
|
SpectrogramConfig SpectrogramConfiguration { get; } |
Gets the spectrogram configuration. |
int LatentChannels { get; } |
Methods
| Method | Summary |
|---|---|
Tensor<T> GenerateAudio(string prompt, string negativePrompt, double durationSeconds, int numInferenceSteps, double? guidanceScale, int? seed) |
Generates audio directly from a text prompt. |
Tensor<T> GenerateSpectrogram(string prompt, string negativePrompt, double durationSeconds, int numInferenceSteps, double? guidanceScale, int? seed) |
Generates a spectrogram from a text prompt. |
Tensor<T> InterpolateStyles(string promptA, string promptB, double alpha, double durationSeconds, int numInferenceSteps, double? guidanceScale, int? seed) |
Interpolates between two music styles. |
Tensor<T> SpectrogramToAudio(Tensor<T> spectrogram) |
Converts a spectrogram to audio waveform. |