PretrainedTokenizerModel

Enums · AiDotNet.Tokenization.Configuration

Specifies pretrained tokenizer models available from HuggingFace Hub.

How It Works

For Beginners: These are industry-standard tokenizers that have been trained on large text corpora. Each is designed for different use cases:

  • BERT models: Best for understanding text (classification, Q&A, NER)
  • GPT models: Best for text generation
  • RoBERTa: Improved BERT with better training
  • T5: Versatile text-to-text model
  • DistilBERT: Faster, smaller BERT

Fields

Field Summary
PretrainedTokenizerModel AlbertBaseV2 ALBERT Base v2 - A Lite BERT with parameter sharing.
PretrainedTokenizerModel BertBaseCased BERT Base Cased - Preserves case information.
PretrainedTokenizerModel BertBaseUncased BERT Base Uncased - The default choice for most NLP tasks.
PretrainedTokenizerModel BertLargeCased BERT Large Cased - Large model preserving case.
PretrainedTokenizerModel BertLargeUncased BERT Large Uncased - Larger model with better accuracy.
PretrainedTokenizerModel CodeBertBase CodeBERT Base - BERT for programming languages.
PretrainedTokenizerModel DistilBertBaseCased DistilBERT Base Cased - Distilled BERT preserving case.
PretrainedTokenizerModel DistilBertBaseUncased DistilBERT Base Uncased - Distilled BERT (40% smaller, 60% faster).
PretrainedTokenizerModel ElectraBase Electra Base - Efficient pretraining approach (base size).
PretrainedTokenizerModel ElectraSmall Electra Small - Efficient pretraining approach.
PretrainedTokenizerModel Gpt2 GPT-2 - OpenAI’s text generation model.
PretrainedTokenizerModel Gpt2Large GPT-2 Large - Even larger GPT-2 variant.
PretrainedTokenizerModel Gpt2Medium GPT-2 Medium - Larger GPT-2 variant.
PretrainedTokenizerModel GraphCodeBert GraphCodeBERT - Code model with data flow.
PretrainedTokenizerModel MicrosoftCodeBert Microsoft CodeBERT - Multi-language code model.
PretrainedTokenizerModel RobertaBase RoBERTa Base - Robustly optimized BERT.
PretrainedTokenizerModel RobertaLarge RoBERTa Large - Large RoBERTa model.
PretrainedTokenizerModel T5Base T5 Base - Text-to-Text Transfer Transformer (base).
PretrainedTokenizerModel T5Large T5 Large - Text-to-Text Transfer Transformer (large).
PretrainedTokenizerModel T5Small T5 Small - Text-to-Text Transfer Transformer (small).
PretrainedTokenizerModel XlnetBaseCased XLNet Base Cased - Autoregressive pretraining.
int value__