IChunkingStrategy

Interfaces · AiDotNet.Interfaces

Defines the contract for text chunking strategies that split documents into smaller segments.

Known implementations: ChunkingStrategyBase

For Beginners

A chunking strategy is like deciding how to slice a pizza.

Think of different ways to divide a long document:

  • Fixed-size chunks: Cut every 500 words (like equal pizza slices)
  • Sentence-based: Keep sentences together (like cutting between toppings)
  • Paragraph-based: Keep paragraphs intact (like cutting by sections)
  • Semantic: Group related content (like separating different flavor sections)

Why chunk documents?

  • Long documents don’t fit in the AI model (like a pizza too big for one plate)
  • Smaller chunks make search more precise (finding exactly the relevant part)
  • You can retrieve just the relevant sections, not entire documents

For example, searching a 100-page manual:

  • Without chunking: Return the entire manual (overwhelming)
  • With chunking: Return just the 2 paragraphs that answer your question (perfect!)

How It Works

A chunking strategy determines how to divide large text documents into smaller, manageable pieces. This is essential for RAG systems because embedding models have maximum token limits, and smaller chunks enable more precise retrieval. Different strategies balance between preserving context and creating appropriately-sized segments.

Properties

Property Summary
int ChunkOverlap { get; } Gets the number of characters that should overlap between consecutive chunks.
int ChunkSize { get; } Gets the target size for each chunk in characters.

Methods

Method Summary
IEnumerable<ValueTuple<string, int, int>> ChunkWithPositions(string text) Splits a text string into chunks and returns them with position metadata.
IEnumerable<string> Chunk(string text) Splits a text string into chunks according to the strategy’s rules.