AutoTokenizer
Helpers & Utilities · AiDotNet.Tokenization.HuggingFace
AutoTokenizer provides HuggingFace-style automatic tokenizer loading. This class automatically detects and loads the appropriate tokenizer type based on the model configuration.
How It Works
Usage mirrors the HuggingFace transformers library:
Methods
| Method | Summary |
|---|---|
ITokenizer FromPretrained(string modelNameOrPath, string cacheDir) |
Loads a tokenizer from a pretrained model name or path. |
Task<ITokenizer> FromPretrainedAsync(string modelNameOrPath, string cacheDir) |
Asynchronously loads a tokenizer from a pretrained model name or path. |
bool IsCached(string modelName, string cacheDir) |
Checks if a tokenizer is cached locally. |
string GetDefaultCacheDir() |
Gets the default cache directory for tokenizer files. |
string[] ListCachedModels(string cacheDir) |
Lists all cached tokenizer models. |
void ClearCache(string modelName, string cacheDir) |
Clears the cache for a specific model or all models. |