T5TextTokenizer

Models & Types · AiDotNet.Tokenization.Specialized

The tokenizer of the pretrained T5 checkpoints (Raffel et al. 2020): t5-base’s SentencePiece unigram model, ids for the special strings, and an end-of-sequence id, matching sentencepiece and transformers 4.x’s slow T5Tokenizer, so a T5 encoder loaded from those checkpoints reads the ids it was trained on.

How It Works

The vocabulary is the 32,000 SentencePiece pieces (<pad> 0, </s> 1, <unk> 2) and 100 sentinels, <extra_id_N> = 32099 − N. Special strings in the text map to their ids, the text between them is segmented on its own, and </s> is appended unless the ids already end with it.

Constructors

Constructor Summary
T5TextTokenizer() The tokenizer of t5-base (shared by t5-small, t5-large and the T5 v1.0 checkpoints).
T5TextTokenizer(SentencePieceModel model) A T5 tokenizer over another SentencePiece model with T5’s special ids.

Properties

Property Summary
int VocabularySize { get; } The vocabulary size, pieces plus sentinels (32,100 for t5-base).

Methods

Method Summary
int[] Encode(string text) The ids of text, ending with </s>.

Fields

Field Summary
int EosId The end-of-sequence id (1).
int ExtraIds The number of sentinel ids after the SentencePiece pieces.
int PadId The padding id (0).