Playtomic graded eight new models on 39 real Slack questions, with two judges checking code and data
Playtomic ran eight newly released models through its Slack agent Dory on 39 real questions, graded by two judges from different labs. The company says public rankings measure generic skills, so other teams would have to rerun the method on their own questions.
The Engineer · Build desk
Key figures: one question is worth about 2.6 points on a quality score out of 100; Dory handles around 1,000 conversations a month.
| Figure | Value | Note | Claim |
|---|---|---|---|
| Points one question is worth, about | 2.6 points | 23 | |
| Quality score, out of | 100 points | 20 | |
| Dory a month, around | 1,000 conversations | 3 |
What happened
- Dory runs today on OpenAI's GPT-5.6 Terra Pro, which is one of the eight models in the test, so the incumbent was scored against its challengers.
- Each judge first derived the correct answer from code, docs and tickets, then graded every answer as correct, partial, wrong, unsupported or unanswered.
- Quality is one point per correct answer and half a point per partial, averaged across both judges, out of 100.
Why it matters
- constraint At 39 questions each one moves a model's score by about 2.6 points, so a gap of a few points between two models is one or two questions.
- exposure Half the field shares a lab with a judge. Playtomic says averaging cancels most self-preference, and any residual bias would sit in the OpenAI and Anthropic models.
- cost Copying the method means 312 answers and at least 624 judge gradings, plus a recorded trace of every step, for a single round of eight models.
- decision Mistral Large 4 fails Playtomic's Zero Data Retention requirement, so its score is information only and cannot make it deployable.
The sample is the first design decision worth copying. Playtomic drew conversations at random, kept the usual mix of question types and removed personal details [8]. It had thumbs-up and thumbs-down ratings and chose not to filter on them. Keeping only well-rated conversations would have favoured Terra, which already answers those well [9]. The final set is 39 questions [8]. If the mix held at two in three data questions, that is about 26 data questions and 13 others [24].
That size sets the resolution. On a 100-point scale where a correct answer earns one point, one question is worth about 2.6 points [23]. A lead of two or three points between models can be a single question going the other way.
The timing is the second decision. Because Playtomic's data shifts from day to day, every model took each question in immediate succession, against identical data [11]. Each ran inside the production agent with the same tools and instructions, read-only, and only the model changed [10].
The judging is where the effort went. Before grading, the judge establishes the correct answer from the code, documentation and tickets. On data questions it then examines the query the model wrote, its metrics and definitions, and whether the figures it quoted agree with what the query returned [14]. Length, tone and confidence earn no points [14].
Playtomic then tested the judges. A second pass with shuffled letters largely agreed with the first [15]. A third judge that compares answers in pairs was secretly given identical pairs and called every one a tie [17]. Where the two judges disagreed, humans spot-checked, and in every case the judge's verdict was as good as or better than the manual check [18]. Of the two, GPT-5.6 Sol grades more harshly, yet both judges put the models in an identical ranking [19].
One gap remains in the bias control. Each judge is slightly more generous to its own lab's models, and Playtomic says averaging the two cancels most of that [16]. Four of the eight models come from OpenAI or Anthropic, the two judge labs [22]. Gemini, GLM, DeepSeek and Mistral have no same-lab judge.
For the scores to transfer, another team's agent would need traffic like Dory's. That means mostly data lookups, code reading and chains of many tool calls under read-only access [4][2]. Public rankings measure maths, self-contained coding and trivia, and Playtomic says a model can top a leaderboard and still be mediocre at Dory's job [26].
The published text we reviewed stops in the latency section, before any scores, so this piece does not report which model won [21].
What to watch
- The scores, typical and slowest-10% latency, and cost per answer for each of the eight models, judged against the bar Playtomic set before seeing any answer.
- Whether Mistral Large 4 gains Zero Data Retention, which is the condition for it to be a candidate for Dory at all.
- Whether Playtomic publishes per-judge scores, which would show how much of the ranking moves for OpenAI and Anthropic models when each judge is taken alone.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence58
- Adoption30
- Hype gap−10
- Incentives
- Insufficient
- Confidence60
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Playtomic tested eight new AI models in Dory, its internal AI agent, and had two AI judges from different companies check every answer against the code and the data.
- [2]
Dory lives in Slack. It queries Looker, reads code on GitHub, checks Linear, Notion and Datadog, and replies in the thread.
- [3]
Dory handles around 1,000 conversations a month, and a single answer can take dozens of steps.
- [4]
About two in three of Dory's questions are data questions; the rest are product behaviour questions answered by reading the code, and knowledge and operations questions.
- [5]
Dory currently runs on GPT-5.6 Terra Pro, from OpenAI.
- [6]
The eight models tested were OpenAI's Terra Pro and GPT-6 Luna Pro, Anthropic's Claude Sonnet 5.5 and Claude Haiku 5.5, Google's Gemini 3.8 Flash, GLM 5.3, DeepSeek V4 Pro and Mistral Large 4.
- [7]
Mistral Large 4 does not yet offer Zero Data Retention, Playtomic's requirement that the provider keeps nothing it sends, but Playtomic tested it anyway.
- [8]
Playtomic drew a random sample of real conversations, keeping the usual mix of question types and removing personal details. 39 questions made the final set.
- [9]
Keeping only the well-rated (thumbs up) conversations would have favoured Terra, which Playtomic already knows answers those well.
- [10]
Every model ran inside the production Dory with the same tools and instructions, and could only read, never change anything. Only the model was changed.
- [11]
Playtomic's data changes daily, so all the models answered each question back to back, against the same data.
- [12]
Playtomic defined what 'as good as the current model' means before seeing a single answer.
- [13]
The two judges were Claude Opus 5.5, from Anthropic, and GPT-5.6 Sol, from OpenAI, each working without seeing the other's verdicts.
- [14]
For every question the judge first works out the right answer from the code, documentation and tickets. For data questions it checks each model's query, metrics and definitions, and whether the quoted figures match what the query returned. It grades each answer as correct, partial, wrong, unsupported or unanswered. Length, tone and confidence earn no points.
- [15]
Answers were anonymous, labelled A, B, C in random order, and a second pass with the letters shuffled checked whether position affected the grade. The two passes largely agreed.
- [16]
Each judge is slightly more generous to its own lab's models, and averaging both judges cancels most of that, according to Playtomic.
- [17]
In a control test, a third judge that compares answers in pairs was secretly given identical pairs and called every one a tie.
- [18]
Humans spot-checked answers where the judges disagreed; the judge matched or improved on the manual check every time.
- [19]
GPT-5.6 Sol is the stricter judge, but both judges rank the models in the same order.
- [20]
The quality score is one point per correct answer and half a point per partial one, averaged across both judges, out of 100.
- [21]
Playtomic looked at three things in order of importance to Dory's users: whether the answer is right, how long it takes (typical time and the slowest 10%), and what it costs.
- [22]
Four of the eight models tested (Terra Pro, Luna Pro, Sonnet 5.5, Haiku 5.5) come from OpenAI or Anthropic, the labs that supply the two judges; the other four have no same-lab judge.
- [23]
With 39 questions and one point per correct answer on a 100-point scale, one question is worth about 2.6 points.
- [24]
If the sample kept the two-in-three data share, about 26 of the 39 questions are data questions and 13 are not.
- [25]
Eight models answering 39 questions gives 312 answers, and two judges grading each gives at least 624 gradings before the shuffled second pass.
- [26]
Public rankings measure generic skills such as maths problems, self-contained coding and trivia. Dory has to read a hurried Slack question, decide where to look, chain many tools, make sense of Playtomic's own data and code, and reply briefly and correctly. Playtomic says a model can top a leaderboard and still be mediocre at that.
Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.
Sources
1 independent publisher whose own reporting we read for this story.
- dev.toWhich model should Dory run on? We tested eight on real questions
1 article · October 13, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.