T5TextTokenizer
Models & Types · AiDotNet.Tokenization.Specialized
The tokenizer of the pretrained T5 checkpoints (Raffel et al. 2020): t5-base’s SentencePiece unigram model, ids for
the special strings, and an end-of-sequence id, matching sentencepiece and transformers 4.x’s slow
T5Tokenizer, so a T5 encoder loaded from those checkpoints reads the ids it was trained on.
How It Works
The vocabulary is the 32,000 SentencePiece pieces (<pad> 0, </s> 1, <unk> 2)
and 100 sentinels, <extra_id_N> = 32099 − N. Special strings in the text map to their ids, the text
between them is segmented on its own, and </s> is appended unless the ids already end with it.
Constructors
| Constructor |
Summary |
T5TextTokenizer() |
The tokenizer of t5-base (shared by t5-small, t5-large and the T5 v1.0 checkpoints). |
T5TextTokenizer(SentencePieceModel model) |
A T5 tokenizer over another SentencePiece model with T5’s special ids. |
Properties
| Property |
Summary |
int VocabularySize { get; } |
The vocabulary size, pieces plus sentinels (32,100 for t5-base). |
Methods
| Method |
Summary |
int[] Encode(string text) |
The ids of text, ending with </s>. |
Fields
| Field |
Summary |
int EosId |
The end-of-sequence id (1). |
int ExtraIds |
The number of sentinel ids after the SentencePiece pieces. |
int PadId |
The padding id (0). |