Six of the eight frameworks in Microsoft's agent-hooks demo have no conformance claim on file
A dev.to author running Microsoft's agent-hooks conformance kit got 69 passes, 4 skips and 0 failures on its reference harness. That covers one host, while the launch demo names eight frameworks and the claims table has three rows.
The Engineer · Build desk
Key figures from the conformance run: 4 vectors skipped, AH-CTK-110 to AH-CTK-113, covering incremental mediation, for the missing incremental_output capability; 73 vectors ran in the kit; 0 vectors failed.
| Figure | Value | Note | Claim |
|---|---|---|---|
| Vectors skipped, incremental mediation | 4 vectors | missing capability: incremental_output | 4 |
| Vectors the kit ran | 73 vectors | 1 | |
| Vectors failed | 0 vectors | 1 |
What happened
- According to the write-up, the launch post says a $840 refund against a $500 cap runs on eight frameworks and that each one 'produces the identical 20-row decision stream'.
- Six of those eight, LangGraph, the OpenAI Agents SDK, Semantic Kernel, LlamaIndex, CrewAI and the Claude Agent SDK, have nothing filed in the claims table.
- A grep of the agent_hooks package for those framework names returns zero hits, and the only Python host binding shipped is create_agent_hooks_middleware for agent_framework.
- The four skipped vectors, AH-CTK-110 to 113, cover incremental mediation and were skipped because the reference harness did not declare the incremental_output capability.
- The repo's CLAIMS.md defines 'conformant' as passing 100% of the CTK vectors applicable to a host's declared surface.
Why it matters
- cost Until a maintainer files a row, a team on one of the six unfiled frameworks writes and tests its own harness adapter before it can cite a conformance claim.
- exposure A host that never declares incremental_output never runs the vectors for a deny arriving after streamed content, so a clean pass leaves that path untested.
- decision A buyer has to read the declared surface on a claims row, because 100% of applicable vectors and 100% of all vectors are different bars.
The kit is easy to reproduce. It has no CLI. It is a Python module: call `load_vectors()`, then `run_vectors(ReferenceHarness, vectors)` [3]. On agent-hooks-sdk 0.1.0b2 and Python 3.13 the author's run covered 73 vectors [1]. It took 0.37 seconds and needed no model, network or API key [2].
The skips come from a capability gate. A host declares what it can do, and vectors outside that declaration are skipped, not failed [5]. Count only the vectors that apply and the run is 69 of 69 [13]. The four skips are 5.5% of the kit [14].
The author's objection is to where those four fall. According to the write-up, they are the vectors that answer what happens to a deny that arrives after content has already streamed to the user [11]. CLAIMS.md requires a host with incremental mediation to declare incremental_output, "so the streaming/incremental vectors run against that discipline instead of being skipped" [12].
So the 69 is a number about one host. For it to transfer to a LangGraph loop, someone would have to write the four-method harness for LangGraph [10]. They would then have to run the same vectors with incremental_output declared and get a clean pass. Nothing in the material says anyone has.
The launch sentence and the table are different kinds of claim. The author does not call the launch post dishonest. The author wrote: "The demo is Microsoft's code on eight hosts; the conformance claim is a per-host artifact someone has to file in a PR" [17]. The demo shows the guard code travels between hosts [18]. The table shows which hosts have been tested against it. By the author's count, six of the eight named frameworks, 75%, have no row [15].
Microsoft's repo invites the filing. It says: "If you maintain a framework and want the conformance report with your name on it, the harness interface is four methods, and we'll do the integration work with you." [10] The author notes that six of the eight have not taken the offer [8]. Every figure here is one author's run and the table as it stood at that run. The write-up gives no date for it.
What to watch
- Whether any maintainer of the six unfiled frameworks files a claim row using the four-method harness.
- Whether a host declares incremental_output and publishes results for AH-CTK-110 through 113, the deny-after-streaming vectors.
- Whether the SDK moves past 0.1.0b2 with a host binding beyond create_agent_hooks_middleware.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence62
- Adoption25
- Hype gap+35
- Incentives
- Insufficient
- Confidence58
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
On agent-hooks-sdk==0.1.0b2 and Python 3.13, the conformance kit ran 73 vectors: 69 passed, 4 skipped and 0 failed.
- [2]
The run took 0.37 seconds and needed no model, network or API key.
- [3]
The CTK has no CLI; it is a Python module called through load_vectors and run_vectors with the ReferenceHarness.
- [4]
The four skipped vectors, AH-CTK-110 to AH-CTK-113, cover incremental mediation and were skipped with the reason missing capabilities: ["incremental_output"].
- [5]
The kit is capability-gated: a host declares what it can do, and vectors outside that declaration are skipped rather than failing.
- [6]
conformance/CLAIMS.md defines 'conformant' as passing 100% of the CTK vectors applicable to the host's declared surface.
- [7]
The repo's claims table has three rows as of the author's run.
- [8]
Six of the eight frameworks named in the launch post (LangGraph, the OpenAI Agents SDK, Semantic Kernel, LlamaIndex, CrewAI, the Claude Agent SDK) have nothing filed in the claims table, so the repo's offer of integration help is unaccepted by six of the eight.
- [9]
A grep across the agent_hooks package for langgraph, langchain, openai, crewai, llama_index, semantic_kernel and claude returns zero hits; the only host binding shipped in Python is create_agent_hooks_middleware, in agent_framework.
- [10]
The repo says: 'If you maintain a framework and want the conformance report with your name on it, the harness interface is four methods, and we'll do the integration work with you.'
- [11]
The four skipped vectors are the ones that answer what happens to a deny that arrives after content has already been streamed to the user.
- [12]
CLAIMS.md requires a host with incremental mediation to declare incremental_output 'so the streaming/incremental vectors run against that discipline instead of being skipped'.
- [13]
Counting only applicable vectors, the run is 69 passes out of 69.
- [14]
The four skipped vectors are about 5.5% of the 73-vector kit.
- [15]
Six of the eight frameworks named in the launch post, 75%, have no claim filed.
- [16]
The launch post says the same support scenario, an $840 refund against a $500 cap, runs across eight frameworks (LangGraph, the OpenAI Agents SDK, Microsoft Agent Framework, Semantic Kernel, LlamaIndex, CrewAI, the Claude Agent SDK and a bare reference host), and that every one of them 'produces the identical 20-row decision stream'.
- [17]
The author wrote: 'The demo is Microsoft's code on eight hosts; the conformance claim is a per-host artifact someone has to file in a PR'.
- [18]
Microsoft wrote a demo suite wired to eight hosts, and the interception contract is host-agnostic enough that the guard code travels.
Sources
1 independent publisher whose own reporting we read for this story.
- dev.to8 Frameworks in the Demo, 3 Rows in the Conformance Table: I Ran Microsoft's agent-hooks CTK
1 article · October 13, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.
Topics
- AI agent frameworksFollow
- Conformance and regression testingFollow
- Agent Governance and GuardrailsFollow