Build1 publisher3 min readPublished
A popularity fallback put off-genre recommendations on 1,724 of 16,311 game pages
The model writing each page's recommendations only ever saw the candidates the code handed it, and when similarity matching found nothing that list was the most-rated games in the catalog. A probability model now scores the candidates first.
The Engineer · Build desk
What happened
- Black Myth: Wukong turned up on one in every ten pages on likethisgame.com that had recommendations, including pages for a music game, a sports game and a puzzle game.
- Each request handed the writing model about six similar games from the catalog plus twenty picked by code, and when the code found no commonality it filled those twenty by user-rating count.
- An audit of 16,311 pages found 1,724 that recommended one of those popular games on a page from an unrelated genre.
- Jev, a decision model from TypeSafe, now rates the candidate games and the writing model works from a list built out of those ratings; Jev does not write the recommendation text.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint The writing model could only choose among the twenty candidates the code supplied, so swapping in a better writer would not have fixed a single page.
- cost Rating candidates adds a second model call per page on top of the call that writes the copy, and because the billing is on input tokens, the scoring bill grows with how much game description every page has to carry.
- decision Once candidates carry scores, someone has to set the threshold and accept the consequence: a page either runs short of recommendations or keeps a weak one.
The fallback is a sort order. The code looked first at what two games had in common, and when it found no match it took the games with the most user ratings [5]. Those twenty went into the prompt alongside about six similar games from the catalog, so roughly three quarters of what the writing model saw was recent releases [4][2].
"When I dug into why, it turned out my own code was steering the model toward those games," the operator of likethisgame.com wrote on dev.to [8]. Across the 16,311 pages checked, the 1,724 off-genre recommendations work out to about 10.6 percent [7][1]. Black Myth: Wukong alone sat on the pages of 12 music games and 10 puzzle games, with Helldivers 2 and Balatro showing the same pattern [6].
Jev, the model now doing the rating, is a decision model from TypeSafe: you send text plus questions and it answers with probabilities instead of prose [9]. Three question types are available. A `noul` returns the probability that a condition holds, a `choice` returns a probability per option, and a `score` returns a rating on a scale you define, such as low, medium and high [11]. The post describes the thing as "an if that can read" [20].
A `noul` of 0.5 does not mean two games are moderately similar; it means that, from the text supplied, yes and no are about equally likely [12]. The post says: "getting a number back does not by itself mean the answer is reliable; you have to try it on your own examples" [13].
The call goes to `https://openrouter.ai/api/alpha/decisions` with model `typesafe/jev-1.13`, a `state` object holding the source game and the candidate array, and one question per candidate keyed `c0`, `c1` and so on [15]. Each game carries its name, release year, genres and description, and the reply exposes `answers.c0.noul` for the first candidate and `usage.cost` for the request [17]. This is a separate endpoint from OpenRouter's chat requests, and you pay for the tokens you send while the answer itself is not billed [14]. Batching therefore removes 19 duplicate copies of the source game's description compared with one request per candidate [3].
Twenty candidates in one context also means each rating is produced next to the other nineteen. The post says it checked whether asking about twenty games at once changes the result and sent the same games one at a time; the text available ends mid-sentence before that comparison [18]. Its contents list promises sections on filtering low-scoring recommendations, on what the change cost, and on what is still unsolved [19]. The work was done with small side-by-side tests, some changes kept and some dropped [21].
What to watch
- Whether the batch-of-twenty ratings match the one-at-a-time ratings on the same games; the post's own comparison is where the published evidence stops.
- A recount of the 16,311 pages after the rebuild, since the off-genre total is the only measure of whether candidate scoring worked.
- What threshold the filter ends up using, and how many pages fall below the minimum number of recommendations as a result.