HuggingFaceTokenizerLoader

Helpers & Utilities · AiDotNet.Tokenization.HuggingFace

Loads HuggingFace pretrained tokenizers.

Methods

Method Summary
ITokenizer LoadFromDirectory(string modelPath) Loads a HuggingFace tokenizer from a directory.
ITokenizer LoadFromHub(string modelName, string cacheDir) Loads a tokenizer from HuggingFace Hub by model name.
ITokenizer LoadFromTokenizerJson(string tokenizerJsonPath) Loads a tokenizer from a tokenizer.json file.
Task<ITokenizer> LoadFromHubAsync(string modelName, string cacheDir) Asynchronously loads a tokenizer from HuggingFace Hub.
void SaveToDirectory(ITokenizer tokenizer, string outputPath) Saves a tokenizer to HuggingFace format.