MidiTokenizer
Models & Types · AiDotNet.Tokenization.Specialized
MIDI tokenizer for symbolic music representation using configurable tokenization strategies.
Inheritance: TokenizerBase → MidiTokenizer
For Beginners
Think of this tokenizer as a translator between musical notes and a language that machine learning models can understand. Just like how text tokenizers convert words into numbers, this tokenizer converts musical notes into tokens.
Imagine a piano: each key has a pitch (which note), and when you press it, you control the velocity (how hard), duration (how long), and timing (when in the song). This tokenizer captures all these aspects in different ways:
- REMI: Like detailed sheet music notation - captures everything (pitch, velocity,
duration, timing) as separate tokens. Use this when you need to preserve all musical details. Example output: [“Bar”, “Position_0”, “Pitch_60”, “Velocity_16”, “Duration_4”]
- CPWord: Like a condensed notation - combines note information into single tokens.
Use this when you want a smaller vocabulary for your model. Example output: [“Bar”, “TimeShift_2”, “Note_60_16_4”]
- SimpleNote: Like a basic melody line - just pitch and duration, no dynamics.
Use this for simple melody generation or when velocity doesn’t matter. Example output: [“Pitch_60”, “Duration_4”, “Rest_2”, “Pitch_64”, “Duration_4”]
How It Works
This tokenizer converts MIDI music data into sequences of tokens that can be used for training machine learning models on music generation, classification, or analysis tasks. It supports three tokenization strategies, each offering different trade-offs between expressiveness and vocabulary size.
Constructors
| Constructor | Summary |
|---|---|
MidiTokenizer(IVocabulary vocabulary, SpecialTokens specialTokens, TokenizationStrategy strategy, int ticksPerBeat, int numVelocityBins) |
Creates a new MIDI tokenizer with the specified configuration. |
Methods
| Method | Summary |
|---|---|
List<string> Tokenize(string text) |
Tokenizes text representation of MIDI. |
List<string> TokenizeNotes(IEnumerable<MidiNote> notes) |
Tokenizes a list of MIDI notes using the configured strategy. |
MidiTokenizer CreateCPWord(SpecialTokens specialTokens, int ticksPerBeat, int numVelocityBins) |
Creates a MIDI tokenizer with CPWord (Compound Word) strategy. |
MidiTokenizer CreateREMI(SpecialTokens specialTokens, int ticksPerBeat, int numVelocityBins) |
Creates a MIDI tokenizer with REMI (Revamped MIDI) strategy. |
MidiTokenizer CreateSimpleNote(SpecialTokens specialTokens, int ticksPerBeat) |
Creates a MIDI tokenizer with SimpleNote strategy. |