Submitting a task starts a short pipeline. Both agents get the same sandbox, the same action vocabulary and the same step budget, their positions are randomized, and vendor names are scrubbed before anyone votes [7]. The design leans on one rule more than any other. A battle whose visible output reveals which model produced it is thrown out of the rating calculation [8]. That is the correct way to defend a blind vote. It is also a filter that does not sample evenly, since the models it penalises are the ones with a habit of introducing themselves.
The thesis underneath came out of the founders' previous product. Coasty was a computer-use agent aimed at legacy desktop software and third-party portals [4], and it reported 82.81% on OSWorld Verified, a suite spanning browsers, office software and operating-system tools [5]. Jannu and Kovuru say building against that suite is what showed them the decay path: agents improve, developers optimize against the published list, and items work their way into training data [6].
The failure statistics deserve arithmetic. About a third of runs do not complete [1]. If the two agents in a battle failed independently of each other, both would fail in roughly 11.6% of battles [2]. The launch post reports about 7% [16], which sits 4.6 points below that baseline [3]. Task difficulty should push the observed figure above the independent product rather than below it, because a hard task tends to defeat both entrants. So either matchmaking is skewing which pairs meet, or completing a task and surviving a human judgement are being counted as different things. No battle counts and no per-model denominators were published [17], so a reader cannot settle which.
Whether any of these rates transfer depends on the incoming stream. The tasks are whatever users bring, such as researching flights or assembling an expense report [3], which means the test set is chosen by the people who wanted a free agent rather than by a model provider [9]. That is the attraction for a lab. It is also the caveat for anyone reading the ranking as a general capability measure: the sampling frame is one product's user base.
What is actually being sold is trace length. The public sample stops at voted battles and the first three actions of each run, while commercial tiers keep longer traces and additional battle metadata [11]. Three actions show that an agent opened a browser and typed something, but not where it got lost, the part a model developer would pay for. The corpus and the ranking are also governed separately: three earlier Coasty battles, one of them voted, sit in the underlying record while staying off the leaderboard [19]. Metering by trace length is defensible engineering. It also makes the free ranking the sales demo for the dataset.
