Harsher tool-schema changes broke Strands, LangGraph and CrewAI in three different ways over 36 runs

The rename that passed 9/9 in all three frameworks in the author's earlier comparison ^[4] turned out to be the easy rung of four. "One step up the ladder, and all three frameworks die in three different ways," the author wrote ^[13]. The rest of the setup stayed fixed. Every run used the same model and the same recording proxy ^[1]. The task prompt asked for an 80 to 120 word digest checked with word_count and did not change by a character between levels ^[3]. Only the tool's signature moved ^[3].

The split starts with who fills in the arguments. In the LangGraph build, the verify node calls word_count(state["draft"]) as ordinary Python, so a changed signature raises a TypeError on that line ^[9]. Each run reached the draft-generation LLM call and died at verification ^[9]. "Silent failure effectively does not exist in LangGraph," the author wrote. "In exchange, no run ever produces output." ^[15]

In Strands and CrewAI the model reads the schema and chooses the arguments. A mismatch becomes something the model has to interpret. At the type level the Strands model sent a numeric document id, got "document not found" back, wrote the digest anyway and exited 0 with zero verifications ^[6]. Mean token use rose from 2,505 at the rename level to 6,004, about 2.4 times ^[7][2]. According to the post, no error surfaced ^[6][7]. At the remove level the argument-less tool returned an 8-word placeholder, and the Strands model accepted it as a passed check in 3 of 3 runs ^[8]. CrewAI believed the same placeholder and exited 0 ^[11].

CrewAI broke on the way back at the type level. Its model put digest text into the int field, 9 wrong-arg calls across 3 runs ^[10]. The tool answered with a JSON parse error, and the provider rejected the next request, the one carrying that error, with a 400 ^[10].

The add level, a required "note" argument the prompt never mentions ^[2], looked like the harshest change. Strands and CrewAI invented a plausible value in 6 of 6 runs and went back to verifying ^[12]. An invented value satisfies the schema. It shows only that the model can fill a field. The portion of the post available does not report what those resumed checks returned.

Of the 27 runs above the rename level ^[1], 9 were LangGraph crashes ^[9] and 6 were add-level runs carried by an invented argument ^[12]. The other 12, Strands and CrewAI at the type and remove levels, ended in a silent exit 0 or a provider 400 ^{[3][6][8][10][11]}.

Those counts come from one task, one model and three runs per cell ^[1]. For them to carry over, your tools would have to fail the way this one did, returning "document not found" or a quiet placeholder as a normal result ^[6][8]. A tool that raises on bad input is a different experiment.

The harness is the part worth copying. It scores a run as a silent failure when the process exits 0 but the word count was never verified ^[5]. It also counts wrong-arg calls, tool-layer rejections and whether the model recovered ^[5]. The repo includes all code, traces and analysis scripts ^[14].