MT3<T>
Models & Types · AiDotNet.Audio.MusicAnalysis
MT3 (Multi-Track Music Transcription) model using T5-style encoder-decoder architecture.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → MT3<T>
Implements: IMusicTranscriber<T>
For Beginners
MT3 listens to a full song with multiple instruments and writes out the sheet music (as MIDI) for each instrument separately. It can tell which notes the piano is playing while also transcribing the guitar, drums, and bass at the same time.
Usage:
How It Works
MT3 (Gardner et al., 2022, Google) is a Transformer-based model that transcribes polyphonic audio into MIDI across multiple instruments simultaneously. It uses a T5-style encoder-decoder architecture with spectrogram input and tokenized MIDI output, achieving state-of-the-art multi-instrument transcription on the Slakh2100 dataset.
Constructors
| Constructor | Summary |
|---|---|
MT3(NeuralNetworkArchitecture<T> architecture, MT3Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates an MT3 model in native training mode. |
MT3(NeuralNetworkArchitecture<T> architecture, string modelPath, MT3Options options) |
Creates an MT3 model in native training mode. |
Properties
| Property | Summary |
|---|---|
int MidiOffset { get; } |
|
int NumMidiNotes { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<TranscribedNote<T>> ExtractNotes(Tensor<T> frameActivations, Tensor<T> onsetActivations, double frameThreshold, double onsetThreshold) |
|
IReadOnlyList<TranscribedNote<T>> Transcribe(Tensor<T> audio) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<IReadOnlyList<TranscribedNote<T>>> TranscribeAsync(Tensor<T> audio, CancellationToken cancellationToken) |
|
Tensor<T> GetFrameActivations(Tensor<T> audio) |
|
Tensor<T> GetOnsetActivations(Tensor<T> audio) |