HuggingFaceTokenizerLoader
Helpers & Utilities · AiDotNet.Tokenization.HuggingFace
Loads HuggingFace pretrained tokenizers.
Methods
| Method | Summary |
|---|---|
ITokenizer LoadFromDirectory(string modelPath) |
Loads a HuggingFace tokenizer from a directory. |
ITokenizer LoadFromHub(string modelName, string cacheDir) |
Loads a tokenizer from HuggingFace Hub by model name. |
ITokenizer LoadFromTokenizerJson(string tokenizerJsonPath) |
Loads a tokenizer from a tokenizer.json file. |
Task<ITokenizer> LoadFromHubAsync(string modelName, string cacheDir) |
Asynchronously loads a tokenizer from HuggingFace Hub. |
void SaveToDirectory(ITokenizer tokenizer, string outputPath) |
Saves a tokenizer to HuggingFace format. |