IMusicSourceSeparator<T>

Interfaces · AiDotNet.Interfaces

Interface for music source separation models that isolate individual instruments/vocals from a mix.

Implements: IFeatureAware, IFullModel<T, Tensor<T>, Tensor<T>>, IGradientComputable<T, Tensor<T>, Tensor<T>>

Known implementations: BSRoFormer<T>, BandSplitRNN<T>, DannaSep<T>, HTDemucs<T>, MelBandRoFormer<T>, MusicSourceSeparator<T>, SCNet<T>

For Beginners

Source separation is like un-mixing a smoothie back into its original fruits.

How it works:

  1. The mixed audio is converted to a spectrogram
  2. A neural network learns which parts belong to which source
  3. Masks are applied to isolate each source
  4. Individual spectrograms are converted back to audio

Common separations:

  • 2-stem: Vocals vs Accompaniment
  • 4-stem: Vocals, Drums, Bass, Other
  • 5-stem: Vocals, Drums, Bass, Piano, Other

Use cases:

  • Karaoke (remove vocals)
  • Remixing (isolate and rearrange parts)
  • Music transcription (analyze individual instruments)
  • Sample extraction (get drum loops, vocal hooks)
  • Music education (practice with isolated parts)

Popular models:

  • Demucs (Facebook/Meta)
  • Spleeter (Deezer)
  • Open-Unmix

How It Works

Music source separation (also called audio source separation or “unmixing”) takes a mixed audio signal and separates it into individual components like vocals, drums, bass, and other instruments.

This interface extends IFullModel for Tensor-based audio processing.

Properties

Property Summary
IReadOnlyList<string> SupportedSources { get; } Gets the sources this model can separate.
bool IsOnnxMode { get; } Gets whether this model is running in ONNX inference mode.
int NumStems { get; } Gets the number of stems/sources this model produces.
int SampleRate { get; } Gets the expected sample rate for input audio.

Methods

Method Summary
SourceSeparationResult<T> Separate(Tensor<T> audio) Separates all sources from the audio mix.
Task<SourceSeparationResult<T>> SeparateAsync(Tensor<T> audio, CancellationToken cancellationToken) Separates all sources asynchronously.
Tensor<T> ExtractSource(Tensor<T> audio, string source) Extracts a specific source from the mix.
Tensor<T> GetSourceMask(Tensor<T> audio, string source) Gets the soft mask for a specific source.
Tensor<T> Remix(SourceSeparationResult<T> separationResult, IReadOnlyDictionary<string, double> sourceVolumes) Remixes the separated sources with custom volumes.
Tensor<T> RemoveSource(Tensor<T> audio, string source) Removes a specific source from the mix.