Twilio scored 2,692 error-code pages with a script before a model touched 1,138

The person reading an error-code page has already lost something. A message failed to send, or a webhook came back with a number they have never seen before. "Error-code pages are the docs you read when something is already going wrong," Twilio's post says ^[7].

The post is direct about the tempting alternative: hand every page to an LLM and say "make these better." "Please don't!" it says ^[19]. What sits in front of the model instead is a plain script, no LLM and no randomness, grading each page out of 100 ^[13]. Structure is worth up to 50 points, for valid frontmatter fields and the expected sections ^[14]. Resources are worth up to 15 ^[16]. Content is worth up to 35, and any page with under 40 characters of body text does not count as having content ^[15]. So 65 of the 100 points describe the page's shape ^[7], and the content test asks whether prose exists. Twilio has open-sourced the reusable core ^[6].

The pass numbers show which pages the model actually saw. Generation finished in about 11 hours, around 34.6 seconds a page ^[11]. At that rate 1,138 pages comes to 39,375 seconds, or 10.9 hours ^[1], while the 2,692 codes counted in the audit ^[8] would have taken about 26 hours ^[2]. The 18,000-hour manual comparison ^[12] works out the same way: at two eight-hour days a page ^[10], 1,138 pages is 18,208 hours, and the full catalog would be nearer 43,000 ^[3]. The updated pages are about 42% of the audited catalog ^[8].

Generation was 11 hours of a project that ran roughly 12 weeks ^[1]. At 40 hours a week, that is 11 hours out of 480, a little over 2% ^[4]. The rest was scoring, reviewing and shipping, plus the checks on the model's output, including a hash on each code sample so it could not quietly rewrite one ^[5]. The post does not report reviewer hours or how many diffs were rejected. At ten minutes a diff, opening 1,138 of them is about 190 hours of someone's month ^[6].

The strongest outcome evidence in the post predates this pipeline. On an earlier set of messaging error codes, Twilio says documentation updates alone drove a 45% decrease in error volume ^[17], and that alongside other changes the error rate for those codes fell from 23% to 8.8% ^[18], a drop of 14.2 points and about 62% in relative terms ^[5]. The second figure carries the other changes with it.

For a team weighing the same pass on its own docs, the split that matters is between pages that are malformed and pages that are wrong. A script can count the first: missing fields, an empty body, no links. The second needs a signal from the people hitting the error, and Twilio picked that idea up from Plaid, whose error docs carry a "Was it helpful?" widget that tells the team where a page is falling short ^[20]. On the content side, the standard Twilio held itself to was one error code, one customer fix ^[21].