PretrainedTokenizerModel
Enums · AiDotNet.Tokenization.Configuration
Specifies pretrained tokenizer models available from HuggingFace Hub.
How It Works
For Beginners: These are industry-standard tokenizers that have been trained on large text corpora. Each is designed for different use cases:
- BERT models: Best for understanding text (classification, Q&A, NER)
- GPT models: Best for text generation
- RoBERTa: Improved BERT with better training
- T5: Versatile text-to-text model
- DistilBERT: Faster, smaller BERT
Fields
| Field | Summary |
|---|---|
PretrainedTokenizerModel AlbertBaseV2 |
ALBERT Base v2 - A Lite BERT with parameter sharing. |
PretrainedTokenizerModel BertBaseCased |
BERT Base Cased - Preserves case information. |
PretrainedTokenizerModel BertBaseUncased |
BERT Base Uncased - The default choice for most NLP tasks. |
PretrainedTokenizerModel BertLargeCased |
BERT Large Cased - Large model preserving case. |
PretrainedTokenizerModel BertLargeUncased |
BERT Large Uncased - Larger model with better accuracy. |
PretrainedTokenizerModel CodeBertBase |
CodeBERT Base - BERT for programming languages. |
PretrainedTokenizerModel DistilBertBaseCased |
DistilBERT Base Cased - Distilled BERT preserving case. |
PretrainedTokenizerModel DistilBertBaseUncased |
DistilBERT Base Uncased - Distilled BERT (40% smaller, 60% faster). |
PretrainedTokenizerModel ElectraBase |
Electra Base - Efficient pretraining approach (base size). |
PretrainedTokenizerModel ElectraSmall |
Electra Small - Efficient pretraining approach. |
PretrainedTokenizerModel Gpt2 |
GPT-2 - OpenAI’s text generation model. |
PretrainedTokenizerModel Gpt2Large |
GPT-2 Large - Even larger GPT-2 variant. |
PretrainedTokenizerModel Gpt2Medium |
GPT-2 Medium - Larger GPT-2 variant. |
PretrainedTokenizerModel GraphCodeBert |
GraphCodeBERT - Code model with data flow. |
PretrainedTokenizerModel MicrosoftCodeBert |
Microsoft CodeBERT - Multi-language code model. |
PretrainedTokenizerModel RobertaBase |
RoBERTa Base - Robustly optimized BERT. |
PretrainedTokenizerModel RobertaLarge |
RoBERTa Large - Large RoBERTa model. |
PretrainedTokenizerModel T5Base |
T5 Base - Text-to-Text Transfer Transformer (base). |
PretrainedTokenizerModel T5Large |
T5 Large - Text-to-Text Transfer Transformer (large). |
PretrainedTokenizerModel T5Small |
T5 Small - Text-to-Text Transfer Transformer (small). |
PretrainedTokenizerModel XlnetBaseCased |
XLNet Base Cased - Autoregressive pretraining. |
int value__ |