ClipTokenizerFactory
Helpers & Utilities · AiDotNet.Tokenization
Factory for creating CLIP-compatible tokenizers.
For Beginners
CLIP needs a special tokenizer to break text into pieces.
A tokenizer factory is like a tool shop that builds tokenizers:
- You can load a pretrained tokenizer (recommended for production)
- You can create a simple tokenizer for testing
- The factory handles all the configuration details
Example usage:
How It Works
CLIP uses a BPE tokenizer with a vocabulary of 49408 tokens. This factory provides methods to create tokenizers from pretrained vocabulary files or to use a default configuration for testing.
Methods
| Method | Summary |
|---|---|
BpeTokenizer CreateShapeCompatibleForTesting(int vocabSize, IEnumerable<string> corpus) |
Creates an offline CLIP test tokenizer whose embedding-table shape is exactly vocabSize. |
BpeTokenizer CreateSimple(IEnumerable<string> corpus, int vocabSize) |
Creates a simple CLIP tokenizer for testing without pretrained files. |
BpeTokenizer FromPretrained(string vocabPath, string mergesPath) |
Creates a CLIP tokenizer from pretrained vocabulary and merge files. |
EncodingOptions GetDefaultEncodingOptions(int maxLength) |
Gets the default encoding options for CLIP text encoding. |
bool IsClipCompatible(ITokenizer tokenizer) |
Validates that a tokenizer is compatible with CLIP. |
Fields
| Field | Summary |
|---|---|
int DefaultMaxLength |
The default maximum sequence length for CLIP text encoder. |
int DefaultVocabSize |
The default vocabulary size for CLIP models. |
string ClipPattern |
The CLIP-specific pre-tokenization pattern. |