Build1 publisher3 min readPublished
Word overlap in one listing line decided every skill invocation across 38 Claude Code runs
A test published on dev.to ran one commit-message skill through Claude Code 38 times, changing only how it was described, and found that "Helps with git stuff." never fired while a file with no frontmatter always did.
The Engineer · Build desk
What happened
- A test published on dev.to made 38 headless claude -p calls against Claude Code 2.1.273 on 2026-09-16, each in a throwaway project.
- The skill body stayed the same in every variant apart from one control, and only the way the skill was described changed between runs.
- The description "Helps with git stuff." never triggered the skill in six runs.
- A specific description, the same vague line with a when_to_use field added, and a SKILL.md with no frontmatter at all were each called six times out of six.
- Under the vague description, Claude answered all six direct and indirect requests itself in one turn without opening the skill file.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- decision The real authoring choice is which phrasings land in the listing entry, so a description has to carry the words a user types even when they duplicate what the skill name already says.
- exposure The failure is silent: the model returns a plausible answer, no error appears in the transcript, and a house-format skill can sit unused until someone reads the commit log.
- constraint The finding is pinned to a listing with a single entry, so it cannot yet say which skill wins when a dozen installed entries all mention git.
- contradiction If a file with no description field can still trigger, then guidance keyed to that field underdescribes the mechanism, and anyone debugging a skill that will not fire may be editing the wrong field.
The decision happens before the skill file is read. Claude Code puts one entry per skill in front of the model, and by the writeup's account the words in that entry are what get matched against the request [6]. A run counted as an invocation only when the transcript carried an assistant `tool_use` block whose name is `Skill` [17]. Until that call fires, the house format in SKILL.md is a file in a directory nothing has opened [9].
The vague line plus a `when_to_use` field fired in all six of its runs, the same count as the carefully written description [5]. The writeup attributes that to word overlap in the single listing line; the field the words lived in did not change the count [6].
The description that fired was written to name the phrases a user would actually type: "Writes git commit messages in this repository's house format (type(scope): summary line, a short why, a Refs line). Use when the user asks for a commit message or wants a diff summarized for the git log." [19] The indirect request was "Give me a one-line summary of this change for the git log." [14] Its wording and the description's second sentence share the phrase "for the git log" [24].
The skills docs say of the `description` field that Claude "uses this to decide when to apply the skill", and the troubleshooting entry for a skill that does not trigger opens with "Check the description includes keywords users would naturally say" [7][8]. The experiment attaches counts to that advice. Main cells ran three times and the smaller controls twice, and the writeup says that with counts this small it does not read the table as rates [20]. Each six-run cell is three direct requests and three indirect ones [26].
For the numbers to transfer, a reader's setup has to look like this one. Every run started in a fresh `mktemp -d` directory holding exactly one skill at `.claude/skills/<name>/SKILL.md` and a project settings file containing `"disableBundledSkills": true` [9]. Settings came from `--setting-sources project` only, and auto memory was off [10]. Each `skill_listing` attachment in the transcripts shows `"skillCount": 1` [11]. The directory was named `kestrel` in all but one variant, so the folder name could not stand in for the description [16]. The model was whatever the CLI picked for the account, which the transcripts name as `claude-opus-5[1m]` [2].
The answers Claude produced under the vague description were not bad. A typical first line was "Add exponential backoff to withRetry and fail fast on AuthError" [23], for a diff in which `withRetry` rethrows an `AuthError` immediately and sleeps with exponential backoff between attempts [25]. It had no `type(scope):` prefix and no `Refs:` line [22]. The skill body ends with `Refs: none`, a string the model would not produce on its own [12], and that marker is what made the miss countable: 22 of the 38 runs called the skill, and the marker appeared in exactly those 22 [18].
What to watch
- A rerun with bundled skills enabled and a dozen entries in the listing, which would show whether overlap still picks the right skill when several lines contain the request's words.
- Any Claude Code release that changes how the listing entry is built, especially for a SKILL.md with no frontmatter.
- Documentation that states what the listing line contains, rather than advice about keywords in the description field.