SentencePieceModel

Models & Types · AiDotNet.Tokenization.Algorithms

A pretrained SentencePiece model read from its .model file (Kudo and Richardson 2018): the normalizer of the model’s precompiled character map, then the unigram model’s Viterbi segmentation (Kudo 2018) or the BPE model’s score-ordered merges (Sennrich et al. 2016) with byte fallback, as the reference library computes them, so a pretrained model sees exactly the ids it was trained on.

How It Works

Normalization follows normalizer.cc: the longest rule of the precompiled double-array trie (darts-clone) rewrites each prefix, leading and repeated whitespace is dropped, a dummy prefix “▁” is added and spaces become “▁”. Segmentation follows unigram_model.cc``EncodeOptimized: the best-scoring path, with a later candidate replacing an earlier one only when strictly better, an unknown node of one character (score: lowest piece score − 10) wherever no single-character piece matches, and runs of unknown pieces merged into one.

BPE follows bpe_model.cc``Encode: the text split into characters, adjacent pairs whose concatenation is a piece merged highest score first (the leftmost on ties), unused pieces split back into the pair that formed them, and, with byte fallback, every unknown character emitted as its UTF-8 bytes’ <0xXX> pieces (sentencepiece_processor.cc).

A character model follows char_model.cc: each Unicode character of the normalized text is a piece (or the longest user-defined symbol starting there), unknown ones merged into runs as above.

User-defined symbols of a unigram model pass through the normalizer unchanged and score bytes × max score − 0.1 in the segmentation, as the reference treats them. BPE models with user-defined symbols, and unigram models with byte fallback, are refused rather than segmented differently from the reference.

Constructors

Constructor Summary
SentencePieceModel(byte[] model) Reads a SentencePiece .model (a serialized ModelProto).

Properties

Property Summary
bool IsBpe { get; } Whether this is a BPE model (otherwise unigram).
int BosId { get; set; } The id of the beginning-of-sentence piece, or −1.
int Count { get; } The number of pieces.
int EosId { get; set; } The id of the end-of-sentence piece, or −1.
int PadId { get; set; } The id of the padding piece, or −1.
int UnknownId { get; } The id of the unknown piece.

Methods

Method Summary
List<ValueTuple<string, int>> EncodeNormalized(string normalized) The pieces and ids of text already in the model’s piece alphabet (spaces as “▁”), with no normalization: what a Hugging Face fast tokenizer’s model step reads after its own normalizer and pre-tokenizer.
List<ValueTuple<string, int>> EncodePieces(string text) The pieces and ids of text.
SentencePieceModel FromGzip(Stream compressed) Reads a gzip-compressed .model.
bool IsControl(int id) Whether a piece is a control symbol (, , …) or a user-defined one, both of which a Hugging Face tokenizer converted from this model matches as added tokens.
bool IsUserDefined(int id)
int PieceToId(string piece) The id of piece, or the unknown id.
int[] Encode(string text) The piece ids of text (no beginning- or end-of-sentence ids).
string Decode(IEnumerable<int> ids) The text of piece ids (control pieces dropped, byte pieces joined into their UTF-8 text, “▁” read as a space, the dummy prefix removed).
string IdToPiece(int id) The piece with id id.
string Normalize(string text) The normalized text the segmentation runs on (normalizer.cc Normalize).