RecursiveCharacterChunkingStrategy
Models & Types · AiDotNet.RetrievalAugmentedGeneration.ChunkingStrategies
Recursively splits text using a hierarchy of separators to preserve document structure.
Inheritance: ChunkingStrategyBase → RecursiveCharacterChunkingStrategy
For Beginners
This is a smart splitter that keeps related text together.
Think of it like organizing a document by trying the best splits first:
Priority 1: Split by double newlines (paragraphs) “Paragraph 1\n\nParagraph 2\n\nParagraph 3” → Keeps each paragraph whole
Priority 2: If paragraphs are too big, split by single newlines (sentences/lines) “Long paragraph with\nmultiple lines\nthat need splitting” → Splits at line breaks
Priority 3: If lines are too big, split by periods (sentences) “First sentence. Second sentence. Third sentence.” → Splits at sentences
Priority 4: If sentences are too big, split by spaces (words) “This is a very long sentence without periods” → Splits at words
Priority 5: Last resort, split by characters “ReallyLongWordWithNoSpaces” → Splits anywhere
Why this is better than simple splitting:
- Keeps paragraphs together when possible (best semantic unity)
- Falls back gracefully when content is too large
- Preserves natural document structure
- Works well with various document formats (code, articles, books)
Example with chunkSize=100, overlap=20:
Input: “First paragraph.\n\nSecond paragraph that is very long and needs to be split into multiple chunks.\n\nThird paragraph.”
- Try splitting by “\n\n” → Second paragraph too large
- Split second paragraph by “ “ → Gets multiple chunks
- Add overlap between chunks
Result:
- Chunk 1: “First paragraph.”
- Chunk 2: “Second paragraph that is very long and” (overlap from chunk 1)
- Chunk 3: “very long and needs to be split into” (overlap from chunk 2)
- Chunk 4: “split into multiple chunks.”
- Chunk 5: “Third paragraph.”
How It Works
This advanced chunking strategy tries to split text using the most semantically meaningful separators first (e.g., double newlines for paragraphs), falling back to less meaningful separators (single newlines, spaces) only when necessary. This preserves the natural structure of documents and keeps related content together.
Constructors
| Constructor | Summary |
|---|---|
RecursiveCharacterChunkingStrategy(int chunkSize, int chunkOverlap, string[] separators) |
Initializes a new instance of the RecursiveCharacterChunkingStrategy class. |