Build1 publisher2 min readPublished
Storing a license ruling as a directional from-into pair stops a model inverting the verdict
License Referee resolves each npm dependency to an SPDX id and looks up a ruling for the exact dependency-into-project direction. On its 22-question test set, all three configurations agreed on the verdict and one invented its source.
The Engineer · Build desk
What happened
- License Referee takes a package.json, a repo URL or an npm package name, resolves each dependency's declared license from the npm registry, normalizes it to SPDX, and looks up the dependency-into-project pair.
- Compatibility runs one way at a time: Apache-2.0 code may go into a GPLv3 work, while GPLv3 code may not go into an Apache-2.0 work.
- Apache-2.0 is compatible with GPLv3 and incompatible with GPLv2, according to both the Apache Software Foundation and the FSF, so the version decides the answer.
- A 22-question golden set ran through the agent, a keyword-search baseline and a no-retrieval baseline, and one no-retrieval run cited an FSF license-list entry that does not exist.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint A badge cannot answer the question this schema is keyed on, because a lookup needs both licenses, their exact versions and a direction; a checker that stores one license string per package has no key to query with.
- contradiction All three configurations reached the correct verdict on the case the submission highlights, so what the golden set demonstrates is citation discipline, and the claim that structure produces better answers is not what the run shows.
- exposure Before the verification pass, a commercial project querying the dataset about a copyleft dependency in its tree would have hit the one direction with no rulings in it.
- decision Anyone wiring this into CI has to decide what unknown does there: the agent declines to guess, so an unmapped pair either fails the build or waits for a human to rule on it.
Start with the schema. A `compatibilityRuling` document stores `from` and `into` as separate fields, so a model reading one cannot silently invert the direction of a ruling [7]. The same record holds the verdict, the combination kind, a verbatim quote, the source URL, and an optional `dissent` field for the cases where the authorities do not agree [6].
The OSI approves licenses the FSF does not call free software, and CC0 is FSF-recommended for software but was withdrawn from OSI review [4]. That is what the `dissent` field is for. The `license` type keeps OSI and FSF flags as separate fields, next to the SPDX id, the category, and the permissions, conditions and limitations [5].
Retrieval is split across two endpoints. `groq_query` and `schema_explorer` run against a GROQ endpoint over the dataset for exact, deterministic verdict lookups; `initial_context` and `knowledge_base_read` run against the Knowledge Base for explanations, obligations, and cross-source reconciliation [9].
For the 22-question result to mean anything about a particular repository, that repository's dependency licenses have to fall inside the 138 rulings in the set [15], and its questions have to resemble the 22 [11]. Anything outside comes back unknown [10]. The submission is a Sanity Challenge entry and the evaluation is the author's own [21].
The typed schema keeps a model from inverting a ruling. A human entered one backwards anyway: CC-BY-4.0 into GPL was stored as incompatible when the FSF's position is the opposite. It stayed that way until an adversarial pass checked every ruling's quote and source URL against the live page it cited [14]. That pass covered FSF, GNU, Apache, Mozilla, Eclipse, OSI and choosealicense.com [18]. It also found 16 quotes that were paraphrased rather than verbatim [14]. The set then grew from 70 entries to 138 [15], which is 68 added, close to double [16]. Sixteen paraphrases against the 70 entries then in the set is roughly one quote in four [17].
The pass also found no rulings at all in one direction: a copyleft dependency going into a permissive or closed-source project [14]. The submission calls that "the single most consequential direction for a real npm tree" [19].
On the fabricated citation, the submission argues that a right verdict resting on an invented authority is worse than a right verdict that honestly reports no ruling exists [13].
The posted demo input is a GPL-2.0-only project depending on `@grpc/grpc-js` at `^1`, and the agent explains why that combination is not allowed and what would fix it [20].
What to watch
- Whether the 138-entry ruling set covers the license spread of a real npm tree, or whether unknown stays the common answer.
- Whether the golden set grows past 22 questions with per-configuration results published, so retrieval can be scored on accuracy and not only on provenance.
- Whether the live quote check becomes a scheduled re-run, since authority pages move and a verbatim quote can go stale.