Compare chunking strategies without changing multiple variables

If chunking, embeddings, query rewriting and reranking change together, a better score cannot identify the cause. Compare one factor at a time before testing interactions.

Declare the controlled experiment

The retrieval evaluation lab uses authored outcomes for fixed-window and heading-aware chunkers.

from retrieval_cases import chunking_comparison_case

result = chunking_comparison_case()
assert result["changed_variables"] == ["chunker"]
assert result["controls"] == {
    "corpus": "same-v1", "questions": "eval-v1", "retriever": "bm25-v1", "k": 3
}
assert result["selected"] == "heading"

Heading-aware has fixture recall@3 of 0.80 versus 0.72 and similar chunk count. These are authored outcomes, not an executed retrieval benchmark.

Freeze the full pipeline

Use identical source bytes, extraction, relevance labels, eligibility, retriever parameters, *k* and evaluation code. Version both chunk manifests. Compare source-span coverage, recall, redundant results, chunk count, tokens and downstream support.

Inspect per-query transitions. Heading-aware chunks may help prose but hurt long tables; fixed windows may preserve local terms yet split definitions. Report slices instead of selecting from one mean.

Avoid hidden changes

Changing chunk size and overlap simultaneously tests a package, not either effect. That can be useful later, but name it. Re-embedding is necessary when chunk text changes; keep the embedding model/config fixed.

Choose on development data and evaluate once on a release set. Preserve negative results and implementation cost. The simpler method can win if quality is equivalent within uncertainty.

The Generative & Agentic AI course uses controlled chunk comparisons before RAG release reports.

Exercise

Compare two chunkers on 50 labelled questions while hashing every controlled input. Produce a transition table and list any accidental second variable discovered during review.

Continue learning

This article is part of the Retrieval quality and grounded answers sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Measure the effect of stale documents on answer quality.
• Continue with Build a retrieval regression suite.

Reference: LangChain text splitters.