Build1 publisher3 min readPublished
Megapixel99's assay tool flags duplicate functions by running each on the same fixed inputs
Megapixel99's assay scan pairs duplicate functions whose outputs match on one fixed input ladder, a check it can apply to roughly a tenth of functions. A match means only that no probe split the pair, so the tool fails the run and hands it to a person.
The Engineer · Build desk
What happened
- In the codebase the tool grew out of, the scan paired is_wordy with _word, a match no textual or name-based detector would make.
- Adding ½, é and a tab plus newline to the inputs split that pair, because isalnum counts ½ as alphanumeric and the other function's isalpha-plus-digit check did not.
- Probing is limited to module-level, undecorated functions of one to three arguments that touch no files, network, clock or randomness; methods and generators are skipped.
- The package installs as assay-checks from pip or npm, with the JavaScript version using the same assay command, as in assay scan src/.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- capability Duplicate discovery no longer waits for someone to suspect a pair, so copies with unrelated names and different code can surface on the first scan.
- constraint A clean report vouches for about one function in ten, so a tree built from classes or I/O-bound code gets little assurance from it.
- decision Anyone adopting it has to check the probe ladder against their own input domain, because a missing rung turns a real difference into a false 'same'.
Differential testing already checks whether two functions agree, but only for a pair somebody declared in advance [3]. assay scan skips the declaration. It walks each comparable function through one fixed, ordered list of probe values and records the return value or the exception type at every position [4]. Functions whose outcome vectors match land in the same hash bucket, so finding candidates takes one pass over the functions with no sweep over every pair [5]. It never reads function names [5].
The ladder is where the method is weakest, and the author, who publishes the code as Megapixel99/assay-checks [1], words the verdicts to match. A "differs" verdict comes with a witness input a stranger can replay [9]. A "same" verdict means only that no rung of a finite ladder told the two functions apart [9]. The tool fails the run on "same" for that reason: it is the weaker claim, and a person has to read the pair [9]. The author wrote that the lesson is worth stealing whatever you test with: "ask what characters your inputs never contain, then add them." [10]
Most of the engineering is in the guard against false matches. The author wrote that every clause exists because of a mistake it now blocks [11]. Two functions that raise TypeError on every input agree perfectly [11]. So do two that return the same constant [11]. Without a guard, the scan would call every one-argument function everyone else's twin [11], a result with perfect recall and no use.
Counting distinct outcomes did not fix it [12]. One returned value plus one exception counts as two outcomes, and that rewards a probe that found a function's type errors and never reached its behaviour [12]. Comparing whole vectors against the identity failed too. A transform whose vocabulary the ladder lacks is the identity wherever it answers and raises everywhere else, so the comparison has to be about the positions where it answered [12]. Then two unrelated query-parameter transforms matched on every rung. The ladder held no key either one recognised, and both copied the object through [13]. A copy is now rejected alongside the identity [13].
Aliasing is the other route to a false pair. A CommonJS module whose export is a function arrives under two keys, and a barrel module hands back the objects its dependencies defined [14]. One helper then appears as registry.js::truncate and as truncate.js::default [14]. Object identity is the correct test for this, and assay uses it to drop those [14]. Names and source text play no part, so a function genuinely copied into two files is still two objects and still counts as two implementations [14].
The author's coverage figure [16] leaves about nine functions in ten outside every verdict [1]. That figure comes from someone else's workload. It carries over only to a codebase with a similar share of functions that pass the probe rules [15]. I'd expect a class-heavy or I/O-heavy tree to come in lower. I think the census is the best decision in the tool, and the author calls it the design position they would defend hardest [16]. Each scan ends with counts of what it refused and why, files and functions as separate populations, with probed plus not probed equal to the function count [16].
What to watch
- Any widening of the probe rules to methods or to functions that take an injected clock, the change that would push coverage past a tenth.
- Census and verdict counts from scans of codebases other than the one the tool grew out of.
- Whether the default ladder ships with non-ASCII and whitespace rungs like the ones that split is_wordy from _word, or leaves adding them to each user.