RepoExec grades a passing solution on whether it called the helper in its own prompt

Dependency Invocation Rate is a set intersection over identifiers. Dg is the set of identifiers in the generated code, Ds is the set of dependencies extracted from the reference solution, and the score is |Dg n Ds| / |Ds| ^[3]. The retreat from match metrics is deliberate. BLEU weighs every token the same, so a valid alternative implementation loses points for wording, while DIR looks only at the identifiers that name dependencies ^[22].

It cannot tell you the call was right. GPT-4o-mini scores 74.75 DIR against 30.29 pass@1 ^[12], a gap of 44.46 points ^[2], and the paper is explicit that DIR counts whether a name appeared and not whether it was used correctly ^[13]. The two families fail in opposite ways: pretrained models write correct code that rewrites what was already there, and instruction-tuned models find the dependency and then wrap it in machinery nobody asked for ^[14].

In the Tornado example, _create_future() builds a Future and then strips the extra asyncio debug stack entries the wrapper itself introduces ^[19]. The reference solution calls it. The pretrained model constructs Future() directly, and the tests pass ^[20]. The instruction-tuned version checked isinstance(x, Future) instead of the supplied is_future(), added an isawaitable branch that is nowhere in the spec, and passed as well ^[21].

The suites themselves were LLM-generated, then filtered for syntax and execution and pushed to an average of 99.45 test cases and 96.25% line coverage per problem ^[7]. Strengthening them cost more than five points of pass@1 ^[10]. The paper's diagnosis is that the newly failing solutions had ignored the supplied context and solved the natural-language description instead, breaking at the edges where the repository's own validation helpers live ^[11].

Context comes at three levels: Full is signature, docstring and body, Medium drops the body, and Small is the signature alone ^[15]. Ordered by BasePrompt pass@1, the result is Full, then Small, then Medium, every time ^[16]. The explanation is prompt shape. A signature-plus-docstring dependency looks exactly like the target function's own prompt, so the model reads the run as few-shot examples with the last one blank ^[17]. Under Medium context, over 31% of StarCoder2's generations are empty function bodies ^[18].

For any of this to transfer, the setting has to match yours. Prompts here average 362.92 tokens with dependency bodies and 253.05 with the bodies stripped ^[8], so the bodies are worth about 110 tokens, roughly 30% of the prompt ^[3]. 22.8% of problems cross a file boundary ^[9], and each problem supplies a single level of dependencies ^[5]. The helper is already in the window. In a real repository the retriever decides what Ds contains, and a helper that was never retrieved cannot show up in a DIR score at all.

The best pass@1 across the 18 models is 42.57% ^[4] on 355 problems ^[6], which leaves roughly 204 failures for the strongest entry ^[1]. What the paper measures is those two things: whether the code passes, and whether the supplied dependency was called ^[2]. The maintenance cost is the dev.to write-up's argument, and it puts it this way: "The bill arrives six months later, when you change the function that was supposed to be the only implementation and the change does not take everywhere" ^[23].