The chunker sets the ceiling: how a character boundary turns a correct policy into a wrong answer

Retrieval did its job in that refund example, which is exactly why the bug survives review. The chunk handed to the model is about sale items and contains the phrase the user asked about ^[4], so anyone auditing the retrieval step marks it relevant. The clause that reverses the rule sits in a chunk that scored lower and was never sent ^[5]. Every component reports success and the answer is still wrong ^[6].

That is a different class of failure from the ones a pipeline is instrumented for. A missing document shows up as an empty result set. A decapitated clause shows up as a confident sentence.

The reason you cannot buy your way out with a bigger window is in the embedding itself: a chunk collapses to one point standing for the average meaning of everything inside it ^[8]. The post suggests 200 to 500 tokens as a starting band for prose documentation ^[12], which means the default teams admit to, 500 tokens ^[1], already sits at the ceiling of the recommended range ^[3]. Go to 2,000 tokens, four times that ceiling ^[2], and a chunk spanning refunds, shipping and warranties lands at a midpoint that a narrow question is only near-ish, where a tighter and less relevant chunk can outrank it ^[10]. Three of those in a prompt is 6,000 tokens spent to answer a question about one sentence ^[11]. Go small instead and you get sharp retrieval with dangling pronouns: "It must be unopened" carries nothing once "it" was named a paragraph earlier ^[9].

Overlap is the standard hedge, repeating the last 10 to 20 percent of the previous chunk so a straddling sentence survives whole somewhere ^[13]. Price it properly. Fifteen percent on a 500-token chunk is 75 duplicated tokens in every chunk ^[1], re-embedded and re-stored on each ingestion run ^[14], and the duplicates retrieve as a pair, so one slot in the top k holds a copy ^[15]. The author's own verdict is that this patches arbitrary cuts without making the cuts less arbitrary ^[16].

The alternative is not a cleverer algorithm. Documents arrive with boundaries their authors already declared: headings, sections, paragraphs, list items, table rows ^[17]. A splitter that honours them is doing less work than one counting characters, not more.

What the post pushes to its final section is the measurement method ^[18], and that is where teams will stall. Standard retrieval scoring rewards the truncated chunk, because by every lexical and semantic measure it is on topic. Until an eval separates a chunk that is about the question from a chunk that is sufficient to answer it, chunk size remains a matter of taste with a policy commitment attached.