LanguageModelTokenizerFactory
Helpers & Utilities · AiDotNet.Tokenization
Factory for creating tokenizers appropriate for different language model backbones.
For Beginners
Each language model was trained with a specific tokenizer. Using the wrong tokenizer will produce garbage results. This factory creates a basic tokenizer with the correct special tokens for each model type.
For production use, you should load the actual pretrained tokenizer from
HuggingFace using AutoTokenizer.
How It Works
Different language models use different tokenization schemes:
- OPT, Chinchilla: GPT-style BPE tokenization
- Flan-T5: T5-style SentencePiece tokenization
- LLaMA, Vicuna, Mistral: LLaMA-style SentencePiece tokenization
- Phi, Qwen: GPT-style BPE with custom vocabulary
Methods
| Method | Summary |
|---|---|
ITokenizer CreateForBackbone(LanguageModelBackbone backbone, IEnumerable<string> corpus, int vocabSize) |
Creates a tokenizer appropriate for the specified language model backbone. |
SpecialTokens GetSpecialTokens(LanguageModelBackbone backbone) |
Gets the special tokens configuration for a language model backbone. |
string GetHuggingFaceModelName(LanguageModelBackbone backbone) |
Gets the recommended HuggingFace model name for loading a pretrained tokenizer. |