TokenBasedChunkingStrategy

Models & Types · AiDotNet.RetrievalAugmentedGeneration.ChunkingStrategies

Chunks text by TOKEN count rather than character count. ChunkSize and ChunkOverlap are interpreted as tokens.

Inheritance: ChunkingStrategyBase → TokenBasedChunkingStrategy

For Beginners

models “see” text as tokens (~¾ of a word each). This splitter keeps each chunk under a token budget so it always fits the model, instead of guessing with character counts.

How It Works

LLM context windows and embedding limits are measured in tokens, not characters, so token-aware splitting (LangChain TokenTextSplitter, LlamaIndex token node parser) is the correct default for RAG. A token-counter delegate is injectable so callers can plug an exact tokenizer (tiktoken / HF); the built-in default counts whitespace-separated words as an approximate proxy.

Constructors

Constructor Summary
TokenBasedChunkingStrategy(int maxTokens, int overlapTokens, Func<string, int> tokenCounter)