Page-picking model Jev cut wrong-page retrievals from 33% to 3% in Sayora's receptionist test

The baseline filters on one number. Sayora cuts each page into 1,200-character chunks with 200 characters of overlap and embeds them with OpenAI's text-embedding-3-large at 768 dimensions ^[4]. On every caller turn it injects the closest chunks that score above a cosine similarity of 0.36 ^[5]. For the caller whose plumber's repair was still leaking, the closest match was a blog post called How to Find a Reliable Plumber ^[6]. The post is, admittedly, about plumbers. A similarity score measures closeness of topic and wording, and it cannot check that a page answers the question.

The replacement changes two things at once. The unit of retrieval becomes a whole page with a short description ^[10], and the ranker becomes Jev, a decision model from TypeSafe ^[7][9]. Embeddings run over those same descriptions would be the control. Without that run, the 33% to 3% gap cannot be split between the model and the page-level index.

On 168 questions, 3% is about 5 wrong pages and 33% is about 55 ^[18]. Each question moves the rate by about 0.6 points ^[19].

The questions were model-written. Claude Opus 5.5 read the page text, never the descriptions, and wrote questions the way callers talk ^[13]. Claude Haiku 4.5 wrote the descriptions, each with a title, a path, a 30-word summary and a few likely questions, at about 100 tokens a page ^[12]. Sayora also removed pages it said a page picker should never see ^[17]. Individual property listings went because a feed already covers them. Near-identical "electrician in [town]" landing pages went because a coverage list answers "do you cover X?" better than a choice between 31 copies ^[17]. That left 403 pages across the four sites ^[20].

For 3% to transfer, a site needs pages that each cover one topic, which Sayora says describes a small business site ^[16]. A 30-word summary also has to be enough to tell two pages apart. And callers have to ask questions like the ones Opus wrote.

Sayora's post says of the "none of these" option: "That last part turned out to matter more than the ranking." ^[14] The post as supplied cuts off before it reports how often "none" was the right answer, so the size of that effect is not in the record.

The price holds up on a back-of-envelope check. At about 100 tokens a page, a 100-page site is roughly 10,000 tokens of options per lookup. At the listed input rate that is about 0.04 cents ^[21], close to the roughly twentieth of a cent Sayora quotes ^[15].