SentencePieceModel
Models & Types · AiDotNet.Tokenization.Algorithms
A pretrained SentencePiece model read from its .model file (Kudo and Richardson 2018): the normalizer of the
model’s precompiled character map, then the unigram model’s Viterbi segmentation (Kudo 2018) or the BPE model’s
score-ordered merges (Sennrich et al. 2016) with byte fallback, as the reference library computes them, so a
pretrained model sees exactly the ids it was trained on.
How It Works
Normalization follows normalizer.cc: the longest rule of the precompiled double-array trie (darts-clone)
rewrites each prefix, leading and repeated whitespace is dropped, a dummy prefix “▁” is added and spaces become
“▁”. Segmentation follows unigram_model.cc``EncodeOptimized: the best-scoring path, with a later
candidate replacing an earlier one only when strictly better, an unknown node of one character (score: lowest piece
score − 10) wherever no single-character piece matches, and runs of unknown pieces merged into one.
BPE follows bpe_model.cc``Encode: the text split into characters, adjacent pairs whose concatenation is a
piece merged highest score first (the leftmost on ties), unused pieces split back into the pair that formed them, and,
with byte fallback, every unknown character emitted as its UTF-8 bytes’ <0xXX> pieces
(sentencepiece_processor.cc).
A character model follows char_model.cc: each Unicode character of the normalized text is a piece (or
the longest user-defined symbol starting there), unknown ones merged into runs as above.
User-defined symbols of a unigram model pass through the normalizer unchanged and score
bytes × max score − 0.1 in the segmentation, as the reference treats them. BPE models with user-defined symbols,
and unigram models with byte fallback, are refused rather than segmented differently from the reference.
Constructors
| Constructor | Summary |
|---|---|
SentencePieceModel(byte[] model) |
Reads a SentencePiece .model (a serialized ModelProto). |
Properties
| Property | Summary |
|---|---|
bool IsBpe { get; } |
Whether this is a BPE model (otherwise unigram). |
int BosId { get; set; } |
The id of the beginning-of-sentence piece, or −1. |
int Count { get; } |
The number of pieces. |
int EosId { get; set; } |
The id of the end-of-sentence piece, or −1. |
int PadId { get; set; } |
The id of the padding piece, or −1. |
int UnknownId { get; } |
The id of the unknown piece. |
Methods
| Method | Summary |
|---|---|
List<ValueTuple<string, int>> EncodeNormalized(string normalized) |
The pieces and ids of text already in the model’s piece alphabet (spaces as “▁”), with no normalization: what a Hugging Face fast tokenizer’s model step reads after its own normalizer and pre-tokenizer. |
List<ValueTuple<string, int>> EncodePieces(string text) |
The pieces and ids of text. |
SentencePieceModel FromGzip(Stream compressed) |
Reads a gzip-compressed .model. |
bool IsControl(int id) |
Whether a piece is a control symbol ( |
bool IsUserDefined(int id) |
|
int PieceToId(string piece) |
The id of piece, or the unknown id. |
int[] Encode(string text) |
The piece ids of text (no beginning- or end-of-sentence ids). |
string Decode(IEnumerable<int> ids) |
The text of piece ids (control pieces dropped, byte pieces joined into their UTF-8 text, “▁” read as a space, the dummy prefix removed). |
string IdToPiece(int id) |
The piece with id id. |
string Normalize(string text) |
The normalized text the segmentation runs on (normalizer.cc Normalize). |