WordPieceTokenizer
Models & Types · AiDotNet.Tokenization.Algorithms
WordPiece tokenizer implementation. Used by BERT and similar models.
Inheritance: TokenizerBase → WordPieceTokenizer
Constructors
| Constructor | Summary |
|---|---|
WordPieceTokenizer(IVocabulary vocabulary, SpecialTokens specialTokens, string continuingSubwordPrefix, int maxInputCharsPerWord, bool lowercase, bool? stripAccents) |
Creates a new WordPiece tokenizer. |
Methods
| Method | Summary |
|---|---|
List<string> Tokenize(string text) |
Tokenizes text into WordPiece tokens. |
List<string> TokenizeWords(IEnumerable<string> words) |
WordPiece over words another tokenizer produced, as Hugging Face’s WordpieceTokenizer.tokenize runs on each (BertJapaneseTokenizer feeds it MeCab’s words): each split on whitespace, then matched greedily. |
WordPieceTokenizer BertBaseChinese(IEnumerable<string> additionalTokens) |
BERT-base Chinese’s WordPiece tokenizer: the released 21,128-token vocabulary (google-research/bert, Apache-2.0), shipped with the library, with each CJK character its own word, lowercasing and accent stripping. |
WordPieceTokenizer BertBaseFrenchEuropeanaCased() |
dbmdz’s cased French Europeana BERT WordPiece tokenizer (dbmdz/bert-base-french-europeana-cased, MIT): its 32,000-token vocabulary, shipped with the library, neither lowercased nor accent-stripped. |
WordPieceTokenizer BertBaseJapaneseV3() |
The WordPiece half of tohoku-nlp/bert-base-japanese-v3’s tokenizer (Apache-2.0): its 32,768-token vocabulary, shipped with the library, case kept. |
WordPieceTokenizer BertBaseMultilingualUncased() |
Multilingual BERT-base’s uncased WordPiece tokenizer: the released 105,879-token vocabulary (google-research/bert, Apache-2.0), shipped with the library, with each CJK character its own word, lowercasing and accent stripping. |
WordPieceTokenizer BertBaseSpanishWwmUncased() |
BETO’s WordPiece tokenizer (dccuchile/bert-base-spanish-wwm-uncased, CC BY 4.0): its 31,002-token vocabulary, shipped with the library, lowercased with accents kept (its tokenizer_config sets strip_accents false). |
WordPieceTokenizer BertBaseUncased() |
BERT-base’s uncased WordPiece tokenizer (Devlin et al. |
WordPieceTokenizer BertKorBase() |
kykim/bert-kor-base’s WordPiece tokenizer: its vocabulary, shipped with the library, lowercased with accents kept (its tokenizer_config sets strip_accents false, so Hangul stays composed). |
WordPieceTokenizer SimBertChinese() |
SimBERT’s WordPiece tokenizer (WangZeJun/simbert-base-chinese, which EmotiVoice’s style encoder reads): its 13,685-token Chinese vocabulary, a subset of BERT-base Chinese’s, with each CJK character its own word and lowercasing. |
WordPieceTokenizer Train(IEnumerable<string> corpus, int vocabSize, SpecialTokens specialTokens, string continuingSubwordPrefix) |
Trains a WordPiece tokenizer from a corpus. |