Build1 publisher2 min readPublished
Regression-testing kagent agents in agentevals means ignoring ADK's role field
kagent-agentevals keeps 8 of 14 ADK events from a real kagent session when it builds an agentevals trajectory for regression tests. Its author found the role field crediting some agent tool calls to the user, so the converter reads part types and drops runtime adk_ tools.
The Engineer · Build desk
What happened
- A developer writing at webofmike.com built kagent-agentevals, a converter from kagent session records to agentevals trajectories plus a golden-suite runner that exits non-zero on behavior change.
- In a sample session, events 3, 5 and 11 carried the agent's own tool call or result under content.role user, because kagent routes human-in-the-loop confirmation through the user side.
- The ADK runtime injects a tool called adk_request_confirmation to run the approval handshake, so it appears in sessions even though the agent never chose to call it.
- By default the converter drops every tool whose name starts with adk_ and keeps kagent's ask_user, a real tool the agent chose to call.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- exposure Operators who judge kagent agents by reading replies in the UI will not see an answer produced from memory with no tool call; only an assertion on the trajectory surfaces it.
- capability Because the runner exits non-zero, a change in which tools a kagent agent calls can fail a CI job the same way a failing unit test does.
- constraint Golden references written with runtime plumbing left in would fail on any session without a confirmation turn, so references stay portable across sessions only after adk_ tools are filtered.
- decision Teams adopting the converter have to check their own tool names, since the default string-match filter would also drop any custom tool named with an adk_ prefix.
Scoring starts with a format gap. agentevals wants a flat list of OpenAI-format chat messages. kagent records its agents as Google ADK events, one row per event, with the JSON in `event.data` [3][4]. "Getting from one to the other is not a field rename," the author wrote [17]. According to the author, three of the reasons only showed up when the converter ran against real sessions, not when he read the schema [14].
The rows are noisy. ADK serializes every optional field, so a row holds about forty keys set to null around the four that matter [5]. Forty nulls is a lot to scroll past to find one search query. The first thing the tool grew was a summary view, `kagent-evals extract <session-id> --summary`, that lists every event and says why anything was dropped [6]. In the sample session it dropped six events [1]. Each was marked either as having no content or as filtered confirmation plumbing [7].
A naive mapping goes wrong at the role field. Copy `content.role` into the OpenAI `role` field and the agent's tool calls on confirmation turns get credited to the person. Every comparison written after that scores the wrong speaker [9]. "The trajectory looks plausible and is wrong," the author wrote [16].
The fix ignores the role and branches on part type [10]. A `function_call` part becomes an assistant `tool_calls` entry. A `function_response` part becomes a tool message. Only plain text asks who was speaking, and it reads `author`. The author found that field reliable where `content.role` is not [10]. I think this is the right design. The part type means the same thing on every event. `content.role` changes meaning when a human-in-the-loop confirmation round-trip is in flight [8].
The runner's non-zero exit on a behavior change is what makes this a regression test [1]. The text available does not include a golden reference file or a failing run. The claim that the suite catches behavior changes rests on the author's description of the runner. The converter carries over to another team's sessions only if they are recorded the way these were, with confirmations routed through the user side and runtime tools named with the `adk_` prefix [8][13]. The author says the demo repo and every command in it were run before publishing [15].
What to watch
- Publication of a golden reference file and a failing run, showing whether the suite fails on any trajectory difference or tolerates reordered or extra tool calls.
- Whether kagent or agentevals ships a built-in ADK-to-trajectory import that would replace this converter.