TokenBasedChunkingStrategy
Models & Types · AiDotNet.RetrievalAugmentedGeneration.ChunkingStrategies
Chunks text by TOKEN count rather than character count. ChunkSize
and ChunkOverlap are interpreted as tokens.
Inheritance: ChunkingStrategyBase → TokenBasedChunkingStrategy
For Beginners
models “see” text as tokens (~¾ of a word each). This splitter keeps each chunk under a token budget so it always fits the model, instead of guessing with character counts.
How It Works
LLM context windows and embedding limits are measured in tokens, not characters, so token-aware
splitting (LangChain TokenTextSplitter, LlamaIndex token node parser) is the correct default
for RAG. A token-counter delegate is injectable so callers can plug an exact tokenizer (tiktoken /
HF); the built-in default counts whitespace-separated words as an approximate proxy.
Constructors
| Constructor | Summary |
|---|---|
TokenBasedChunkingStrategy(int maxTokens, int overlapTokens, Func<string, int> tokenCounter) |