AgenticChunker

Models & Types · AiDotNet.RetrievalAugmentedGeneration.ChunkingStrategies

Production-ready intelligent chunker that decides where to split text based on semantic boundaries.

Inheritance: ChunkingStrategyBase → AgenticChunker

For Beginners

This is like a smart text splitter that understands content structure.

Think of it like organizing a book:

  • Don’t split in the middle of a sentence or paragraph
  • Keep related ideas together in the same chunk
  • Start new chunks at natural topic boundaries
  • Maintain context with overlapping content

How it works:

  1. Identifies structural elements (paragraphs, sections, lists)
  2. Calculates semantic coherence scores for potential splits
  3. Creates chunks at natural boundaries
  4. Adds overlap for context preservation

Example:

  • Input: Long article about climate change
  • Output: Chunks at section boundaries, keeping introduction separate from data analysis,

solutions separate from problems, etc.

Unlike simple fixed-size chunking:

  • ✓ Respects paragraph boundaries
  • ✓ Keeps related sentences together
  • ✓ Detects topic changes
  • ✓ Preserves document structure

Production features:

  • No external API dependencies
  • Fast heuristic-based topic detection
  • Configurable chunk sizes and overlap
  • Handles multiple document formats
  • Maintains semantic coherence

How It Works

This chunker analyzes text structure to identify optimal split points based on:

  • Paragraph boundaries
  • Topic transitions (detected via sentence similarity)
  • Natural breaks in content flow (headers, lists, code blocks)
  • Semantic coherence within chunks

Constructors

Constructor Summary
AgenticChunker(int maxChunkSize, int overlap, double coherenceThreshold) Initializes a new instance of the AgenticChunker class.