Real-SWE licenses private production codebases to score coding agents on real business tasks

The sample task published in full is an invoice tax fix, and the instruction is a single paragraph of English ^[7]. Behind that paragraph, each business on the platform settles tax its own way: some maintain a rate themselves, some want each invoice priced against the buyer's destination by a tax authority provider, and some collect nothing ^[7]. Pricing a destination means calling the authority with both addresses, the priced lines and the product category the business sells under. The call goes to the sandbox or the production authority, according to the account the business is on. An address the authority refuses has to be reported without stopping the invoice ^[8]. The rate, the tax and the gross belong on the issued invoice. A settled sale is filed back to the authority under that invoice's number so the returns reconcile. Invoices between European parties show both sides' VAT registrations ^[9]. Two of the three modes need no authority call and the third splits by endpoint, which is four paths, and the customer-level exemption overrides all of them ^[16].

The instruction opens with billing reopening on Monday. That is the most realistic detail in it ^[7].

That environment is a NestJS service in TypeScript talking to TaxJar sandbox, TaxJar production and an InfluxDB ledger, with the endpoints handed over as TAX_JAR_URL, PROD_TAX_JAR_URL and INFLUX_URL ^[10]. Across the benchmark the published tool list runs to 18 services, languages and test runners, including a Linear MCP server, Slack, Intercom, Google Drive and ClickUp, and each task exposes only the services its workflow needs ^[11][15].

The authors write that they use native harnesses to reflect how enterprise engineers work in practice, "evaluating model-and-harness combinations rather than models in isolation", and that they used high reasoning for all models ^[6]. For one of these scores to predict anything in your shop, three things would have to hold: the same harness, the same MCP wiring into your own tracker, and a sandbox for the external service. It would have to answer the way TaxJar's does.

The contamination case in the release is structural. Agents "must navigate proprietary systems whose code and solutions aren't available on the public internet", per the release ^[3]. The write-up does not report a result for any model, so the size of the gap between a public-repo score and a private-repo score is not published ^[17]. Screening favoured real companies with substantial usage, strong engineering teams and demanding production workloads. The named examples include a Luma or Partiful competitor with more than 200,000 users and a top-100 App Store ranking, plus a consumer fintech platform processing more than 100,000 bank statements ^[12].

On specificity, the authors describe their prompts as "slightly underspecified, about par with DeepSWE and Terminal Bench, but specific enough to not omit instructions", with any behaviour the verifier requires stated or reasonably discoverable in the codebase ^[13]. Their stated objection to expert-written and synthetic sets is that such tasks "aren't the verbatim, actual tasks that engineers in real companies need to do" ^[14].