Build1 publisher3 min readPublished
Agent workflows often made models worse on CMU's Synthetic Hospital chart benchmark
Carnegie Mellon's Synthetic Hospital benchmark found agent wrappers often lowered scores for 10 models reading 1,268 synthetic multi-visit patient charts. The models found facts in the charts but struggled to combine them, so a scaffold that adds retrieval steps may be working on the wrong weakness.
The Engineer · Build desk
What happened
- Among 10 models, Kimi 2.5 Thinking led diagnosis reconstruction with a severity-weighted F1 of 0.732, with GPT 5.3 second at 0.703.
- The models were fairly good at locating relevant facts in a chart but substantially weaker at combining them into a diagnosis or patient summary.
- Wrapping a model in an AI-agent workflow frequently lowered its performance on the benchmark, according to The Neuron's account of the paper.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- decision Teams adding agent scaffolding to long-record reasoning need a bare-model baseline on the same charts, since in this benchmark the wrapper was often the change that lowered scores.
- constraint Scores come from a teaching-case disease mix, so a hospital cannot use them to forecast performance on its own patient population.
- capability Public code and data with a known complete patient state let outside teams score missed diagnoses against what each patient actually has, a check real charts cannot support.
A real chart makes a weak answer key. It records only what clinicians happened to document, and access to real records is restricted by privacy, ethics and data-use rules [3]. Carnegie Mellon's benchmark avoids both problems by building patients who never existed [1]. The records come from public medical-education material and are coded to ICD-10-CM, SNOMED CT and LOINC [2]. Its builders know the complete patient state each chart is supposed to encode [4]. When a model misses a diagnosis, the patient really has that diagnosis.
These charts are longitudinal but short. Spreading 5,602 encounters across 1,268 patients gives about 4.4 encounters per patient [1]. The explainer describes the real problem as years of records, with key evidence perhaps six visits back [12]. On average, a synthetic patient's history is shallower than that [1].
Ten licensed physicians made 100 blind real-or-synthetic judgments and were right 53% of the time, statistically indistinguishable from chance [5]. For the people building a benchmark, a coin-flip score from doctors is the result they want. If the judgments are treated as independent, the standard error at n=100 is about 5 points, and a 95% interval runs from roughly 43% to 63% [2]. So physicians could not reliably separate the charts. The study was also too small to rule out a modest ability to do so. The authors say the disease mix follows teaching material and is not meant to mirror real epidemiology [6].
No single model dominated the leaderboard [7]. Kimi 2.5 Thinking led diagnosis reconstruction with a severity-weighted F1 of 0.732, ahead of GPT 5.3 at 0.703 [8]. The gap is 0.029 [3].
The agent result comes with a specific split. The models were fairly good at finding relevant facts in a chart. They were substantially weaker at combining those facts into a diagnosis or a useful summary [9]. According to The Neuron's account of the paper, wrapping a model in an agent workflow frequently made performance worse [10]. That account does not describe the agent setup, which models dropped, by how much, or how the severity weights are set.
I think the split points to the first hypothesis to test. Suppose a scaffold spends its extra steps on retrieval: call a tool, read a slice of the chart, choose the next slice. Then it adds work to the stage the models already handle. It also hands the synthesis step fragments of the record where the bare model saw the whole. Checking this needs the paper's agent configuration.
Three conditions would have to hold before this result says anything about a given deployment. The records would need to resemble teaching cases. The task would need to be diagnosis reconstruction or summary. The scaffold would need to resemble the authors' setup. The code and data are public on GitHub [11]. A team can run its own scaffold and its bare model against the same 1,268 patients [2].
What to watch
- The arXiv paper's description of the agent workflow and per-model score changes, especially whether the drops sit in synthesis tasks or retrieval tasks.
- Independent runs of other agent scaffolds against the bare models on the public GitHub data.
- A larger physician realism study, since 100 judgments cannot rule out a modest ability to spot synthetic charts.