WordPieceTokenizer

Models & Types · AiDotNet.Tokenization.Algorithms

WordPiece tokenizer implementation. Used by BERT and similar models.

Inheritance: TokenizerBase → WordPieceTokenizer

Constructors

Constructor Summary
WordPieceTokenizer(IVocabulary vocabulary, SpecialTokens specialTokens, string continuingSubwordPrefix, int maxInputCharsPerWord, bool lowercase, bool? stripAccents) Creates a new WordPiece tokenizer.

Methods

Method Summary
List<string> Tokenize(string text) Tokenizes text into WordPiece tokens.
List<string> TokenizeWords(IEnumerable<string> words) WordPiece over words another tokenizer produced, as Hugging Face’s WordpieceTokenizer.tokenize runs on each (BertJapaneseTokenizer feeds it MeCab’s words): each split on whitespace, then matched greedily.
WordPieceTokenizer BertBaseChinese(IEnumerable<string> additionalTokens) BERT-base Chinese’s WordPiece tokenizer: the released 21,128-token vocabulary (google-research/bert, Apache-2.0), shipped with the library, with each CJK character its own word, lowercasing and accent stripping.
WordPieceTokenizer BertBaseFrenchEuropeanaCased() dbmdz’s cased French Europeana BERT WordPiece tokenizer (dbmdz/bert-base-french-europeana-cased, MIT): its 32,000-token vocabulary, shipped with the library, neither lowercased nor accent-stripped.
WordPieceTokenizer BertBaseJapaneseV3() The WordPiece half of tohoku-nlp/bert-base-japanese-v3’s tokenizer (Apache-2.0): its 32,768-token vocabulary, shipped with the library, case kept.
WordPieceTokenizer BertBaseMultilingualUncased() Multilingual BERT-base’s uncased WordPiece tokenizer: the released 105,879-token vocabulary (google-research/bert, Apache-2.0), shipped with the library, with each CJK character its own word, lowercasing and accent stripping.
WordPieceTokenizer BertBaseSpanishWwmUncased() BETO’s WordPiece tokenizer (dccuchile/bert-base-spanish-wwm-uncased, CC BY 4.0): its 31,002-token vocabulary, shipped with the library, lowercased with accents kept (its tokenizer_config sets strip_accents false).
WordPieceTokenizer BertBaseUncased() BERT-base’s uncased WordPiece tokenizer (Devlin et al.
WordPieceTokenizer BertKorBase() kykim/bert-kor-base’s WordPiece tokenizer: its vocabulary, shipped with the library, lowercased with accents kept (its tokenizer_config sets strip_accents false, so Hangul stays composed).
WordPieceTokenizer SimBertChinese() SimBERT’s WordPiece tokenizer (WangZeJun/simbert-base-chinese, which EmotiVoice’s style encoder reads): its 13,685-token Chinese vocabulary, a subset of BERT-base Chinese’s, with each CJK character its own word and lowercasing.
WordPieceTokenizer Train(IEnumerable<string> corpus, int vocabSize, SpecialTokens specialTokens, string continuingSubwordPrefix) Trains a WordPiece tokenizer from a corpus.