OnsetsAndFrames<T>

Models & Types · AiDotNet.Audio.MusicAnalysis

Onsets and Frames piano transcription model from Google Magenta.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → OnsetsAndFrames<T>

Implements: IMusicTranscriber<T>

For Beginners

Onsets and Frames is a piano-specific music transcriber. It listens to piano music and detects every key press: when each key is pressed (onset), how long it is held (frame), and which key it is (pitch). The output is a list of notes that can be saved as MIDI or displayed as sheet music.

Usage:

How It Works

Onsets and Frames (Hawthorne et al., 2018) jointly predicts note onsets and frame-level activations for automatic piano transcription. The model uses CNN acoustic features with bidirectional LSTMs, trained on the MAESTRO dataset. It achieves ~90% note F1 on piano.

Constructors

Constructor Summary
OnsetsAndFrames(NeuralNetworkArchitecture<T> architecture, OnsetsAndFramesOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates an Onsets and Frames model in native training mode.
OnsetsAndFrames(NeuralNetworkArchitecture<T> architecture, string modelPath, OnsetsAndFramesOptions options) Creates an Onsets and Frames model in native training mode.

Properties

Property Summary
int MidiOffset { get; }
int NumMidiNotes { get; }

Methods

Method Summary
IReadOnlyList<TranscribedNote<T>> ExtractNotes(Tensor<T> frameActivations, Tensor<T> onsetActivations, double frameThreshold, double onsetThreshold)
IReadOnlyList<TranscribedNote<T>> Transcribe(Tensor<T> audio)
ModelMetadata<T> GetModelMetadata()
Task<IReadOnlyList<TranscribedNote<T>>> TranscribeAsync(Tensor<T> audio, CancellationToken cancellationToken)
Tensor<T> GetFrameActivations(Tensor<T> audio)
Tensor<T> GetOnsetActivations(Tensor<T> audio)