MidiTokenizer

Models & Types · AiDotNet.Tokenization.Specialized

MIDI tokenizer for symbolic music representation using configurable tokenization strategies.

Inheritance: TokenizerBase → MidiTokenizer

For Beginners

Think of this tokenizer as a translator between musical notes and a language that machine learning models can understand. Just like how text tokenizers convert words into numbers, this tokenizer converts musical notes into tokens.

Imagine a piano: each key has a pitch (which note), and when you press it, you control the velocity (how hard), duration (how long), and timing (when in the song). This tokenizer captures all these aspects in different ways:

  • REMI: Like detailed sheet music notation - captures everything (pitch, velocity,

duration, timing) as separate tokens. Use this when you need to preserve all musical details. Example output: [“Bar”, “Position_0”, “Pitch_60”, “Velocity_16”, “Duration_4”]

  • CPWord: Like a condensed notation - combines note information into single tokens.

Use this when you want a smaller vocabulary for your model. Example output: [“Bar”, “TimeShift_2”, “Note_60_16_4”]

  • SimpleNote: Like a basic melody line - just pitch and duration, no dynamics.

Use this for simple melody generation or when velocity doesn’t matter. Example output: [“Pitch_60”, “Duration_4”, “Rest_2”, “Pitch_64”, “Duration_4”]

How It Works

This tokenizer converts MIDI music data into sequences of tokens that can be used for training machine learning models on music generation, classification, or analysis tasks. It supports three tokenization strategies, each offering different trade-offs between expressiveness and vocabulary size.

Constructors

Constructor Summary
MidiTokenizer(IVocabulary vocabulary, SpecialTokens specialTokens, TokenizationStrategy strategy, int ticksPerBeat, int numVelocityBins) Creates a new MIDI tokenizer with the specified configuration.

Methods

Method Summary
List<string> Tokenize(string text) Tokenizes text representation of MIDI.
List<string> TokenizeNotes(IEnumerable<MidiNote> notes) Tokenizes a list of MIDI notes using the configured strategy.
MidiTokenizer CreateCPWord(SpecialTokens specialTokens, int ticksPerBeat, int numVelocityBins) Creates a MIDI tokenizer with CPWord (Compound Word) strategy.
MidiTokenizer CreateREMI(SpecialTokens specialTokens, int ticksPerBeat, int numVelocityBins) Creates a MIDI tokenizer with REMI (Revamped MIDI) strategy.
MidiTokenizer CreateSimpleNote(SpecialTokens specialTokens, int ticksPerBeat) Creates a MIDI tokenizer with SimpleNote strategy.