Tokenization
31 public types in this namespace, organized by kind.
Models & Types (15)
| Type |
Summary |
BpeTokenizer |
Byte-Pair Encoding (BPE) tokenizer implementation for subword tokenization. |
CharacterTokenizer |
Character-level tokenizer that splits text into individual characters. |
CodeBertTokenizer |
CodeBERT-compatible tokenizer for program synthesis and code understanding tasks. |
CodeTokenizer |
Code-aware tokenizer that handles programming language constructs. |
MidiNote |
Represents a MIDI note event with timing, pitch, and velocity information. |
MidiTokenizer |
MIDI tokenizer for symbolic music representation using configurable tokenization strategies. |
PhonemeTokenizer |
Phoneme-based tokenizer for speech synthesis (TTS) applications. |
SentencePieceModel |
A pretrained SentencePiece model read from its .model file (Kudo and Richardson 2018): the normalizer of the model’s precompiled character map, then the unigram model’s Viterbi segmentation (Kudo 2018) or the BPE model’s score-ordered mer… |
SentencePieceTokenizer |
SentencePiece tokenizer implementation using Unigram language model. |
SpecialTokens |
Represents special tokens used by tokenizers. |
T5TextTokenizer |
The tokenizer of the pretrained T5 checkpoints (Raffel et al. |
TokenizationResult |
Represents the result of tokenizing text, including token IDs, tokens, and attention masks. |
UnigramTokenizer |
Unigram Language Model tokenizer using probabilistic segmentation. |
Vocabulary |
Manages a vocabulary of tokens and their IDs. |
WordPieceTokenizer |
WordPiece tokenizer implementation. |
Base Classes (1)
| Type |
Summary |
TokenizerBase |
Base class for tokenizers providing common functionality. |
Interfaces (2)
Enums (5)
Options & Configuration (3)
Helpers & Utilities (5)