Microsoft DevRel's AX playbook traces coding-agent mistakes back to docs, skills and MCP tools

"An evaluation can lie to you convincingly," Microsoft's post says, and its examples are specific ^[6]. It reports perfect scores for code that never compiled, and a "used Platform X" check that passed whether or not the agent used Platform X ^[6]. A check that passes on every input measures nothing. The playbook's evaluation model runs from criteria that judge meaning to gates that prove the code runs ^[7]. The run gate is the cheapest layer to build, and it would have caught the first of those two failures.

Microsoft calls writing criteria the hardest step ^[17]. The playbook asks for criteria a judge rules on the same way every time, calibrated before anyone trusts them and versioned when the product changes ^[8]. It warns against letting a model write them ^[8]. I think the ordering is correct. A judge that flips its verdict on a rerun cannot tell you whether last week's docs edit helped, and the playbook wants every proposed change tested as a hypothesis before it ships ^[10].

"A readout tells you what failed while the trajectory tells you why," Microsoft wrote ^[16]. The split I would adopt first sorts failures into three cases, since each calls for its own fix: the agent never loaded the extension, loaded it but never called it, or called it and applied it incorrectly ^[9]. In my view the first two are discovery problems. Rewriting the content of a skill the agent never opened changes nothing. The playbook also catalogs nine failure patterns that Microsoft says recur across every technology it evaluated, each pointing to the surface to inspect first ^[18].

The post's account of a bad generation is blunt: the wrong SDK version, a deprecated authentication pattern, and an agent that "did exactly what its training data told it to do" ^[15]. It adds that a knowledge cutoff "tells you little about what a model knows about your product" ^[3]. If the cutoff is no guide, the prior has to be measured product by product. Every lever the post names ^[4] changes behavior only if the agent reaches it during the session. So the loaded-or-called question comes before any content fix.

The outcome evidence is thinner. Microsoft credits the evaluations with dozens of shipped fixes, including 46 improvements to the Azure Cosmos DB Agent Kit ^[11]. Forty-six is a count of changes. The post does not report how agent pass rates moved after them. The scenarios ran on Microsoft's own products ^[5]. For the results to carry to another platform, its developers' prompts have to resemble the scenarios its team writes, and its agents need a route to its extensions at all. The method does not require a particular evaluation system, and the playbook lists the capabilities to check for in whichever one a team uses ^[13]. Most of the adoption cost is therefore expert time spent on criteria.

Microsoft also released the method as an AX Practitioner skill to install in a coding agent and question during an evaluation ^[14]. Like any skill, it has to be loaded before it can help.