AutoTokenizer

Helpers & Utilities · AiDotNet.Tokenization.HuggingFace

AutoTokenizer provides HuggingFace-style automatic tokenizer loading. This class automatically detects and loads the appropriate tokenizer type based on the model configuration.

How It Works

Usage mirrors the HuggingFace transformers library:

Methods

Method Summary
ITokenizer FromPretrained(string modelNameOrPath, string cacheDir) Loads a tokenizer from a pretrained model name or path.
Task<ITokenizer> FromPretrainedAsync(string modelNameOrPath, string cacheDir) Asynchronously loads a tokenizer from a pretrained model name or path.
bool IsCached(string modelName, string cacheDir) Checks if a tokenizer is cached locally.
string GetDefaultCacheDir() Gets the default cache directory for tokenizer files.
string[] ListCachedModels(string cacheDir) Lists all cached tokenizer models.
void ClearCache(string modelName, string cacheDir) Clears the cache for a specific model or all models.