Build1 publisher3 min readPublished
Salesforce's RAG accuracy fell to ~46% on complex, customer-representative enterprise documents, versus 90% on standard text-retrieval benchmarks
The engineering team's own figures put one pipeline above 90% on clean text retrieval and near 46% on customer-representative files, with the diagnostic trail running back to parsing and chunking rather than to the model.
The Engineer · Build desk

What happened
- Salesforce's engineering team says two quarters of escalations from customers, forward deployed engineers and solution engineers preceded the work on its RAG pipeline.
- The same pipeline exceeded 90% accuracy on standard text-retrieval benchmarks and dropped to roughly 46% against complex, customer-representative enterprise documents.
- The reported failures were structural: financial tables losing headers at page breaks, coverage matrices turning garbled, and retrieval picking an article about the wrong product.
- Debugging each query backward from answer generation to parsing repeatedly pointed at the earliest stages, where complex content was disappearing before retrieval ran.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint If headers vanish at parse time and row relationships at chunk time, a stronger model and a better prompt cannot buy the accuracy back, which removes the two cheapest levers teams reach for first.
- decision A single end-to-end accuracy figure cannot tell you which of six stages to fund, so acting on a number like 46% means paying for per-stage instrumentation before any remediation is scoped.
- exposure The damage surfaces to customers as a fluent answer rather than an error, so the class of defect being described will not appear in exception logs or alerting.
- precedent A vendor engineering team publishing a gap this wide makes any bare above-90% RAG accuracy claim hard to accept without the document mix it was measured on.
Convert both numbers into errors before judging the size of the gap. Above 90% on the text benchmark means fewer than one wrong answer in ten [1]. Roughly 46% on the enterprise set means about 54 wrong in a hundred [1]. That is at least 5.4 times the error rate from the same pipeline [1].
The mechanism Salesforce describes is a chain of local, reasonable decisions. A coverage matrix with merged cells and nested headers runs onto a second page. The parser separates that second page from its headers. The chunker splits the remaining rows at a fixed token boundary. Retrieval returns one of those fragments, and the model receives values whose meaning is already gone [8]. A parser that drops a header does not raise an error; it returns text. Nothing in the fragment announces the loss, so the model produces a plausible reading of numbers it cannot interpret [8]. Headers discarded at parse time and relationships destroyed at chunk time are not recoverable downstream [9].
That is also why retrieval takes blame it has not earned. A result can look wrong because similarity ranking failed, or because it was the best remaining candidate after parsing threw away the correct table [11]. The two look identical in an end-to-end score, which is the argument for walking the pipeline backward: complete context with a wrong answer indicts generation, incomplete context sends you to retrieval, a missing chunk sends you to chunking, and an absent chunk sends you to parsing [6].
Treat the 46% as a claim about Salesforce's corpus, because that is what it is. The post characterises the evaluation set only as complex, customer-representative enterprise documents, and the figure arrives with no stated document count or scoring rule [15]. For it to say anything about your system, your corpus would need a comparable share of the structures the post's own failure story names, and your ingest would need to look like the one in the example: a single deterministic parse tier and fixed-token chunking [8][12]. If your tables live in a database and your PDFs are prose, the number is about someone else's page breaks.
The parsing argument is where the published material thins out. One strategy is unlikely to serve a corpus holding both clean prose and complex visual structure; a deterministic parser is quick on prose and fails on merged cells, nested headers, charts and diagrams [12]. The post begins to weigh sending every page through deeper model-based processing instead, then breaks off before stating the routing rule [13], and the four capabilities it advertises are named in a caption rather than enumerated in the available text [14].
So the transferable artifact here is the ordering, not the score. Running the backward walk costs one stored artifact per stage per query, plus the per-stage checks that decide whether context was complete before you argue about the model. Salesforce reports that when its team did this, the failures kept resolving to the earliest stages [7]. The stated target of clearing 90% on enterprise documents without one stage hiding another's failures is, on this evidence, still a target [4].
What to watch
- Whether Salesforce publishes the evaluation set size, document mix and scoring rule behind the approximately 46% figure.
- The routing rule for deciding which pages get deeper model-based parsing, which the available text stops short of stating.
- Whether the team reports clearing its stated 90% target on enterprise documents, and with which stage-level checks in place.