BuildNot yet confirmed elsewhere1 publisher3 min readPublished
OpenAI's Asana case study prices 1.5 weeks of engineer time against a five-year staffing plan
The two-week Enzyme removal is credible enough, but it is priced against a five-year staffing plan written by people who did not want the project, and Airbnb's published numbers let you check the rate that plan assumed.
The Engineer · Build desk

What happened
- OpenAI's case study says Asana used Codex to remove Enzyme, an outdated testing system, in two weeks, with model and infrastructure costs of about $12,000.
- The plan that figure is compared against was expected to take at least five years and to cost roughly $6 million, according to the same case study.
- OpenAI puts the human input at 1.5 weeks of engineering effort spread across two calendar weeks, after which Enzyme was fully removed.
- Airbnb migrated a 3,500-file Enzyme component test suite in six weeks using LLMs, against an estimate of 1.5 engineering years to do the same work by hand.
- Gergely Orosz reckons the $6M came from assuming an engineer could migrate five to ten tests a day, then multiplying the resulting engineering years by salary.
Why it matters
- cost The published $12,000 is a models-and-infrastructure number, so any per-migration budget copied from it quietly omits the engineer weeks and the human-finished tail the same class of project still needs.
- constraint With one side denominated in budgeted engineer-years and the other in calendar time, the case study cannot support a productivity multiple; it can only show that an unloved plan was never going to be funded.
- capability The transferable part of Airbnb's result is the retry loop and the purpose-built refactor pipeline, which means a team that buys the model without building the harness buys none of the throughput.
- decision Anyone putting a vendor ROI figure in front of a board now has to name who wrote the counterfactual and whether they wanted the project, because that authorship sets the size of the saving.
Enzyme and React Testing Library are not two dialects of the same test. An Enzyme test operates on a component instance; RTL asserts against the rendered DOM, so the test sees the whole rendered page rather than the component it was written for [6]. Assertions have to be re-expressed in terms of what a user can see, and for complex user journeys the two approaches diverge sharply [7]. Asana went from the first to the second [4]. Both sides of the comparison are describing genuinely awkward work.
The units are where it comes apart. The arithmetic behind the $6M is four engineers at circa $300K a year for five years, according to Gergely Orosz's reading of the case study [5], and it multiplies out cleanly to the published figure [16]. That is also about 1,040 engineer-weeks of budgeted labour [17]. The other side of the comparison is 1.5 weeks of engineering effort [3], and the write-up does not say how many engineers those 1.5 weeks cover, which is a curious omission in a comparison this lopsided.
The $12K is model and infrastructure spend, not the cost of the work [2]. Charge the engineer time at the salary the $6M itself assumes and you add roughly $8,650, for about $20,700 all in [18]. Against an advertised gap of $5.99M [19], that is noise either way.
Airbnb's numbers let you test the rate the plan assumed. Its manual estimate was 1.5 engineering years for 3,500 Enzyme component test files [8]. At 250 working days a year, that implies about nine files per engineer-day [14], the top of the five-to-ten range Orosz guesses Asana's planners used [20]. Push Asana's implied 20 engineering years through Airbnb's rate and the plan is sized for roughly 46,700 test files [15]. If Asana's suite is nowhere near that, the five years measured appetite rather than throughput. Orosz puts it plainly: you estimate a project like this when it is work you really do not want to do [13].
Airbnb also shows where the effort actually sits. Its team built loops that retried failed migrations, and once those existed 75% of files went through in four hours [9]. The remaining quarter needed a purpose-built refactor pipeline, which took the total to 97% over four days, with engineers finishing the last 3% by hand over a week [10]. That was March 2025, on Claude 3.7 Sonnet [11]. The bulk was hours; the tail was a week of humans.
Orosz still finds two weeks credible for Asana, a year on from Airbnb [12]. The outcome can stand while the denominator gets audited. Whether a saving like this transfers depends on who authored the counterfactual and whether they wanted the work, the unit both figures are denominated in, and what the smaller figure leaves out. Calendar weeks set against engineer-years fails on the second before you get to the third.
What to watch
- Whether OpenAI or Asana publishes the test file count and the per-engineer-day rate the five-year plan assumed.
- Whether Asana's engineers describe a retry loop or refactor pipeline, since the case study credits the model rather than a harness.
- Whether the next vendor case study quotes engineer-weeks on both sides of the comparison instead of calendar time on one.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence48
- Adoption58
- Hype gap+52
- Incentives76
- Confidence54
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
OpenAI's case study states the previous staffing plan for the work was expected to take at least five years and was estimated to cost roughly $6 million.
- [2]
OpenAI's case study states that Asana used OpenAI Codex to remove Enzyme, an outdated testing system, in two weeks, with model and infrastructure costs totalling about $12,000.
- [3]
Per OpenAI, Enzyme was fully removed after 1.5 weeks of engineering effort spread across two calendar weeks.
- [4]
Asana migrated from Enzyme to React Testing Library, which the Pragmatic Engineer newsletter says would not be a simple migration.
- [5]
According to Gergely Orosz, OpenAI's arithmetic rests on an estimate of four engineers, each on circa $300K a year, spending five years on the project.
- [6]
Enzyme is oriented towards component testing and operates on a component instance, while React Testing Library operates on the rendered Document Object Model, so the test sees the whole rendered page rather than only the component it was written for.
- [7]
The testing approach differs radically between the two frameworks, especially when testing complex user journeys.
- [8]
Airbnb migrated an Enzyme test suite of 3,500 component test files within six weeks using LLMs; the estimate for doing it by hand was 1.5 engineering years.
- [9]
Airbnb's team built loops to keep retrying migrations, and once they did, 75% of files were migrated in just four hours.
- [10]
Airbnb built a more sophisticated refactor pipeline for the remaining 25% of tests; the new loop migrated 97% of tests in total after running over four days, and engineers finished the remaining 3% with LLM input in a week.
- [11]
Airbnb's LLM-aided migration was in March 2025, when the frontier coding model was Claude 3.7 Sonnet.
- [12]
Orosz writes that on the basis it took Airbnb six weeks, he finds it credible that it took Asana two weeks to migrate probably similarly complex test infrastructure a year later.
- [13]
Orosz writes: "You estimate a project like this when it's an undesirable project you really don't want to do as an engineer!"
- [14]
Airbnb's manual estimate of 1.5 engineering years for 3,500 test files implies about nine files per engineer per working day, assuming 250 working days a year.
- [15]
Twenty engineering years at Airbnb's implied per-file rate would correspond to roughly 46,700 test files.
- [16]
Four engineers at $300,000 a year for five years comes to $6,000,000, matching the figure OpenAI published.
- [17]
The staffing plan behind the $6M represents about 1,040 engineer-weeks of budgeted labour.
- [18]
Adding 1.5 weeks of one engineer's time at the case study's own $300K salary assumption puts about $8,650 on top of the $12,000 model and infrastructure bill, for roughly $20,700 in total.
- [19]
The gap between OpenAI's $6M estimate and the $12,000 actually reported is about $5.99M.
- [20]
Orosz says the time and $6M cost quoted by OpenAI feel inflated, and assumes the numbers came from estimating a maximum of five to ten tests migrated per engineer per day, converting that to engineering years (perhaps 20) and multiplying by the cost of an engineer.
ReportedContestedSource: Gergely Orosz, Pragmatic Engineer2 sources— create a free account to open themView cited source
Sources
1 independent publisher whose own reporting we read for this story.
- blog.pragmaticengineer.comThe Pulse: We need to talk about migrations with AI
1 article · August 27, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.
Topics
- Frontend Testing FrameworksFollow
- Inference cost and context budgetingFollow
- AI-Assisted Code MigrationFollow
- Software Estimation PracticeFollow
- Vendor ROI Case Study ScrutinyFollow