Evaluating LLMs on expert judgment tasks

Frontier models stall in the mid-70s on the triage work Thinking Machines wants to automate

Thinking Machines' own figures show that better prompts carried the models it tested from a coin flip to the mid-70s and then quit, leaving the last five points to an 80% trust bar to be bought with annotation time from the analysts being freed.