MT3<T>

Models & Types · AiDotNet.Audio.MusicAnalysis

MT3 (Multi-Track Music Transcription) model using T5-style encoder-decoder architecture.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → MT3<T>

Implements: IMusicTranscriber<T>

For Beginners

MT3 listens to a full song with multiple instruments and writes out the sheet music (as MIDI) for each instrument separately. It can tell which notes the piano is playing while also transcribing the guitar, drums, and bass at the same time.

Usage:

How It Works

MT3 (Gardner et al., 2022, Google) is a Transformer-based model that transcribes polyphonic audio into MIDI across multiple instruments simultaneously. It uses a T5-style encoder-decoder architecture with spectrogram input and tokenized MIDI output, achieving state-of-the-art multi-instrument transcription on the Slakh2100 dataset.

Constructors

Constructor Summary
MT3(NeuralNetworkArchitecture<T> architecture, MT3Options options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates an MT3 model in native training mode.
MT3(NeuralNetworkArchitecture<T> architecture, string modelPath, MT3Options options) Creates an MT3 model in native training mode.

Properties

Property Summary
int MidiOffset { get; }
int NumMidiNotes { get; }

Methods

Method Summary
IReadOnlyList<TranscribedNote<T>> ExtractNotes(Tensor<T> frameActivations, Tensor<T> onsetActivations, double frameThreshold, double onsetThreshold)
IReadOnlyList<TranscribedNote<T>> Transcribe(Tensor<T> audio)
ModelMetadata<T> GetModelMetadata()
Task<IReadOnlyList<TranscribedNote<T>>> TranscribeAsync(Tensor<T> audio, CancellationToken cancellationToken)
Tensor<T> GetFrameActivations(Tensor<T> audio)
Tensor<T> GetOnsetActivations(Tensor<T> audio)