Tokenization

31 public types in this namespace, organized by kind.

Models & Types (15)

Type Summary
BpeTokenizer Byte-Pair Encoding (BPE) tokenizer implementation for subword tokenization.
CharacterTokenizer Character-level tokenizer that splits text into individual characters.
CodeBertTokenizer CodeBERT-compatible tokenizer for program synthesis and code understanding tasks.
CodeTokenizer Code-aware tokenizer that handles programming language constructs.
MidiNote Represents a MIDI note event with timing, pitch, and velocity information.
MidiTokenizer MIDI tokenizer for symbolic music representation using configurable tokenization strategies.
PhonemeTokenizer Phoneme-based tokenizer for speech synthesis (TTS) applications.
SentencePieceModel A pretrained SentencePiece model read from its .model file (Kudo and Richardson 2018): the normalizer of the model’s precompiled character map, then the unigram model’s Viterbi segmentation (Kudo 2018) or the BPE model’s score-ordered mer…
SentencePieceTokenizer SentencePiece tokenizer implementation using Unigram language model.
SpecialTokens Represents special tokens used by tokenizers.
T5TextTokenizer The tokenizer of the pretrained T5 checkpoints (Raffel et al.
TokenizationResult Represents the result of tokenizing text, including token IDs, tokens, and attention masks.
UnigramTokenizer Unigram Language Model tokenizer using probabilistic segmentation.
Vocabulary Manages a vocabulary of tokens and their IDs.
WordPieceTokenizer WordPiece tokenizer implementation.

Base Classes (1)

Type Summary
TokenizerBase Base class for tokenizers providing common functionality.

Interfaces (2)

Type Summary
ITokenizer Interface for text tokenizers.
IVocabulary Interface for vocabulary management.

Enums (5)

Type Summary
PhonemeSet Supported phoneme sets.
PretrainedTokenizerModel Specifies pretrained tokenizer models available from HuggingFace Hub.
ProgrammingLanguage Programming languages supported by the code tokenizer.
TokenizationStrategy MIDI tokenization strategies that control how musical notes are converted to tokens.
TreeSitterLanguage Supported programming languages for Tree-sitter parsing.

Options & Configuration (3)

Type Summary
EncodingOptions Options for encoding text into tokens.
TokenizationConfig Configuration options for tokenization in the prediction pipeline.
TokenizerConfig Configuration for HuggingFace tokenizers.

Helpers & Utilities (5)

Type Summary
AutoTokenizer AutoTokenizer provides HuggingFace-style automatic tokenizer loading.
ClipTokenizerFactory Factory for creating CLIP-compatible tokenizers.
HuggingFaceTokenizerLoader Loads HuggingFace pretrained tokenizers.
LanguageModelTokenizerFactory Factory for creating tokenizers appropriate for different language model backbones.
PretrainedTokenizerModelExtensions Extension methods for PretrainedTokenizerModel enum.