LanguageModelTokenizerFactory

Helpers & Utilities · AiDotNet.Tokenization

Factory for creating tokenizers appropriate for different language model backbones.

For Beginners

Each language model was trained with a specific tokenizer. Using the wrong tokenizer will produce garbage results. This factory creates a basic tokenizer with the correct special tokens for each model type.

For production use, you should load the actual pretrained tokenizer from HuggingFace using AutoTokenizer.

How It Works

Different language models use different tokenization schemes:

  • OPT, Chinchilla: GPT-style BPE tokenization
  • Flan-T5: T5-style SentencePiece tokenization
  • LLaMA, Vicuna, Mistral: LLaMA-style SentencePiece tokenization
  • Phi, Qwen: GPT-style BPE with custom vocabulary

Methods

Method Summary
ITokenizer CreateForBackbone(LanguageModelBackbone backbone, IEnumerable<string> corpus, int vocabSize) Creates a tokenizer appropriate for the specified language model backbone.
SpecialTokens GetSpecialTokens(LanguageModelBackbone backbone) Gets the special tokens configuration for a language model backbone.
string GetHuggingFaceModelName(LanguageModelBackbone backbone) Gets the recommended HuggingFace model name for loading a pretrained tokenizer.