Vocabulary

Models & Types · AiDotNet.Tokenization.Vocabulary

Manages a vocabulary of tokens and their IDs.

Implements: IVocabulary

Constructors

Constructor Summary
Vocabulary(Dictionary<string, int> tokenToId, string unkToken) Creates a vocabulary from an existing token-to-ID mapping.
Vocabulary(string unkToken) Creates a new vocabulary.

Properties

Property Summary
IReadOnlyDictionary<int, string> IdToToken { get; } Gets the ID-to-token mapping.
IReadOnlyDictionary<string, int> TokenToId { get; } Gets the token-to-ID mapping.
int Size { get; } Gets the vocabulary size.

Methods

Method Summary
IEnumerable<string> GetAllTokens() Gets all tokens in the vocabulary.
bool ContainsId(int id) Checks if a token ID exists in the vocabulary.
bool ContainsToken(string token) Checks if a token exists in the vocabulary.
int AddToken(string token) Adds a token to the vocabulary.
int GetTokenId(string token) Gets the token ID for a given token.
string GetToken(int id) Gets the token for a given token ID.
void AddTokens(IEnumerable<string> tokens) Adds multiple tokens to the vocabulary.
void Clear() Clears the vocabulary and re-adds the unknown token.