ClipTokenizerFactory

Helpers & Utilities · AiDotNet.Tokenization

Factory for creating CLIP-compatible tokenizers.

For Beginners

CLIP needs a special tokenizer to break text into pieces.

A tokenizer factory is like a tool shop that builds tokenizers:

  1. You can load a pretrained tokenizer (recommended for production)
  2. You can create a simple tokenizer for testing
  3. The factory handles all the configuration details

Example usage:

How It Works

CLIP uses a BPE tokenizer with a vocabulary of 49408 tokens. This factory provides methods to create tokenizers from pretrained vocabulary files or to use a default configuration for testing.

Methods

Method Summary
BpeTokenizer CreateShapeCompatibleForTesting(int vocabSize, IEnumerable<string> corpus) Creates an offline CLIP test tokenizer whose embedding-table shape is exactly vocabSize.
BpeTokenizer CreateSimple(IEnumerable<string> corpus, int vocabSize) Creates a simple CLIP tokenizer for testing without pretrained files.
BpeTokenizer FromPretrained(string vocabPath, string mergesPath) Creates a CLIP tokenizer from pretrained vocabulary and merge files.
EncodingOptions GetDefaultEncodingOptions(int maxLength) Gets the default encoding options for CLIP text encoding.
bool IsClipCompatible(ITokenizer tokenizer) Validates that a tokenizer is compatible with CLIP.

Fields

Field Summary
int DefaultMaxLength The default maximum sequence length for CLIP text encoder.
int DefaultVocabSize The default vocabulary size for CLIP models.
string ClipPattern The CLIP-specific pre-tokenization pattern.