Vals sells a private AI exam to the labs it grades

A team choosing between two models this quarter is reading a chart the vendor drew. TechCrunch, in its piece on Vals, states the incentive plainly: good benchmarks pretty much always mean good PR ^[5]. The same piece reports that companies have figured out how to outwit legacy benchmarking systems, many of them older and not built to measure modern models ^[6]. One route is mechanical. A benchmark that publishes its questions can be trained against ^[7].

Krishnan's answer is to keep the questions in-house and to score domain work instead of general knowledge, in law, finance and coding ^[7][8]. "Historically, I think evaluation has been done to evaluate intelligence in a very abstract way," he said, describing bar-exam-style tests ^[11]. The question he says he wants answered instead: "Can they do work that produces a product of the same quality as a human within every domain?" ^[10]

The money runs the opposite way from what a buying team might assume. Model developers pay Vals to test them, and Krishnan likens the arrangement to a student paying the College Board to sit the SAT ^[12]. TechCrunch reports those evaluations are becoming key decision-making factors for companies looking to acquire new AI models ^[13]. So the developer pays for the test, and the tasks inside it stay private from the buyer leaning on the score ^[7][12]. Whether a client can keep a poor result unpublished goes unanswered ^[20].

Labs are buying. Vals started this year with eight people and now has 25, with another 10 to 15 planned ^[15]. That lands at 35 to 40, between four and five times the January headcount, inside twelve months ^[19]. The subject matter is widening into places where an outside grade may be the only grade: Krishnan lists a benchmark on recursive self improvement, plus work in mental health, cybersecurity, biosecurity and the law of armed conflict, testing whether models can apply the Geneva Convention ^[17]. He said the hope is to analyze how, "if these models ran wild in the world, what the negative implications would be" ^[18].

How much weight a benchmark number deserves in a vendor decision comes down to who paid for the test, whether your team can inspect the items, and whether the tasks resemble the work your users actually bring. Vals is paid by the developer, and the items stay in-house ^[12][7]. The match between its tasks and your users' work is the one a deployment team can settle without anyone's permission, by running the last fifty real tickets through both candidates and reading the output.