IVocabulary
Interfaces · AiDotNet.Tokenization.Interfaces
Interface for vocabulary management.
Known implementations: Vocabulary
Properties
| Property | Summary |
|---|---|
IReadOnlyDictionary<int, string> IdToToken { get; } |
Gets the ID-to-token mapping. |
IReadOnlyDictionary<string, int> TokenToId { get; } |
Gets the token-to-ID mapping. |
int Size { get; } |
Gets the vocabulary size. |
Methods
| Method | Summary |
|---|---|
IEnumerable<string> GetAllTokens() |
Gets all tokens in the vocabulary. |
bool ContainsId(int id) |
Checks if a token ID exists in the vocabulary. |
bool ContainsToken(string token) |
Checks if a token exists in the vocabulary. |
int AddToken(string token) |
Adds a token to the vocabulary. |
int GetTokenId(string token) |
Gets the token ID for a given token. |
string GetToken(int id) |
Gets the token for a given token ID. |
void AddTokens(IEnumerable<string> tokens) |
Adds multiple tokens to the vocabulary. |
void Clear() |
Clears the vocabulary. |