Chunk documents around meaningful boundaries

Fixed token windows are simple, but they can join unrelated policies or separate a condition from its exception. Start from document structure, then use size limits within those boundaries.

Preserve section identity

The RAG ingestion lab splits a synthetic text at policy headings.

from ingestion_cases import chunk_boundaries_case

result = chunk_boundaries_case()
assert result["count"] == 2
assert result["headings"] == ["Refund Policy", "Travel Policy"]
assert result["cross_policy_chunk"] is False
print(result["chunks"])

The two chunks keep refund and travel rules separate. The check is small; it does not establish an optimal chunker for arbitrary documents.

Build a hierarchy

Parse document → section → paragraph/list/table → source span. Keep headings with their content and repeat a short hierarchy label in child chunks where needed. Do not split a table row, numbered procedure or definition from its qualifier merely to hit a target size.

Set a maximum using the embedding model’s supported input and the retrieval task. Long sections may need recursive splitting; very short adjacent paragraphs may be merged when they share a semantic unit. Store chunker version, start/end offsets, pages and source digest.

Evaluate retrieval consequences

Create labelled question-to-evidence pairs. Compare section-aware chunks with fixed windows at similar index size. Measure whether the complete supporting span appears in top-*k*, plus duplicate/redundant results and answer support. Inspect failures where the question needs two related sections.

Chunking also affects deletion and access. Every chunk should inherit source authorization and be removable from every index by document/version ID. Rechunk into a new versioned index rather than overwriting active vectors invisibly.

The Generative & Agentic AI course treats chunking as an evaluated ingestion decision connected to retrieval metrics.

Exercise

Implement heading-aware and 300-unit fixed chunkers for five permitted documents. Create 25 evidence labels, compare recall and redundant results, then record which document structures break each method.

Continue learning

This article is part of the RAG ingestion and document preparation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Handle scanned documents with OCR quality checks.
• Continue with Compare chunk overlap with a retrieval benchmark.

Reference: LangChain text-splitter concepts.