Vocabulary
Models & Types · AiDotNet.Tokenization.Vocabulary
Manages a vocabulary of tokens and their IDs.
Implements: IVocabulary
Constructors
| Constructor | Summary |
|---|---|
Vocabulary(Dictionary<string, int> tokenToId, string unkToken) |
Creates a vocabulary from an existing token-to-ID mapping. |
Vocabulary(string unkToken) |
Creates a new vocabulary. |
Properties
| Property | Summary |
|---|---|
IReadOnlyDictionary<int, string> IdToToken { get; } |
Gets the ID-to-token mapping. |
IReadOnlyDictionary<string, int> TokenToId { get; } |
Gets the token-to-ID mapping. |
int Size { get; } |
Gets the vocabulary size. |
Methods
| Method | Summary |
|---|---|
IEnumerable<string> GetAllTokens() |
Gets all tokens in the vocabulary. |
bool ContainsId(int id) |
Checks if a token ID exists in the vocabulary. |
bool ContainsToken(string token) |
Checks if a token exists in the vocabulary. |
int AddToken(string token) |
Adds a token to the vocabulary. |
int GetTokenId(string token) |
Gets the token ID for a given token. |
string GetToken(int id) |
Gets the token for a given token ID. |
void AddTokens(IEnumerable<string> tokens) |
Adds multiple tokens to the vocabulary. |
void Clear() |
Clears the vocabulary and re-adds the unknown token. |