Build1 publisher2 min readPublished
Caret's author handed the spec to the agent that can read the repository
A month and a half of prototyping an experimental functional language with Codex ended with eight artifacts around the repository and a documentation flow in which ChatGPT writes the prompts and Codex writes the spec.
The Engineer · Build desk
What happened
- Asked whether it could prototype a language design idea, the model skipped the yes-or-no and produced a small Java project with a parser, an AST and a few more classes.
- A month and a half on, the Caret prototype is surrounded by specifications, a conformance matrix, an implementation plan, executable examples, project cards, CI released artifacts, tests and agent instructions.
- In the first process, ChatGPT wrote spec text that the author checked against the design and added to the language specification by hand.
- Codex became the owner of the documentation once the author had it update the language specification and README.md, because ChatGPT was unaware of the changes Codex had made.
- A REPL and a rudimentary test suite written in Caret were added so that code could be typed in and executed immediately.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint A specification can only track code if it is written by the party that can see the code, so a chat model outside the repository is limited to producing text that ages from the moment it is pasted in.
- cost The review burden stays with the single maintainer, who has to check each generated spec section against the design before it lands, and the generated prose is written for the implementing agent rather than for a reader.
- capability Tests written in the target language let the human verify a claim of implementation by running it, which a diff review of the agent's own commit cannot give.
- decision Anyone running an agent on a project past prototype size now has to choose where design intent lives, and here it moved out of the chat window into repository files that one model drafts prompts for and another writes.
The second documentation flow has four steps: discuss the idea with ChatGPT, ask ChatGPT to write a prompt for Codex, feed that prompt to Codex, and let Codex add the section to the spec [17]. ChatGPT keeps the part the author rates it for. It grasps ideas in compact form when you are precise, and it points out flaws, ambiguities and underspecified areas, while the solutions it offers back are surface-level and situational [19].
"The solution to AI drift turned out to be not better prompting, but surrounding the project with specific artifacts that tracked its progress and kept the consistency of design in check," the author wrote [4]. The rules file arrived early: on ChatGPT's prompting, an initial AGENTS.md of project instructions went into the repository [8], after the Codex plugin was installed in the IDE [7].
Verbosity, which makes chat output tiresome to read, is what a specification wants [20]. The author added: "What it cranks out is not perfect for a human reader, but it's detailed and precise enough as an instruction for implementation" [18].
The artifacts exist because one pass does not fit. It became clear early that Codex would not implement everything in the specification in one go, because the scope of the project was already too big [10]. In my view the spec, the implementation plan and the conformance matrix hold state between sessions that neither model holds by itself.
Visibility was the other complaint. The author asked how the prototype was even supposed to be used [12], and answered it inside Caret, where unit tests written in the language compare function results against expected values [13]. A claim of "implemented" can then be checked by execution. Getting the REPL to work inside the IDE's terminal took a couple of iterations [14].
A language prototype is close to the best case for this setup. Features are enumerable, examples execute, and a conformance matrix has something concrete to conform to. For the same eight classes of artifact [21] to pay off on another project, the spec has to be checkable by running something. Where it cannot be executed, the files still have to be kept current by hand, unverified.
This is one maintainer's account of one project, published on dev.to [24], and it does not report how often Codex went off track before or after the artifacts appeared [25]. The narrative covers the first two versions of the documentation flow; the conformance matrix, the GitHub project cards and the CI released artifacts appear in the opening inventory of what the project became [3].
What to watch
- The author says the way of working with Codex will probably keep evolving; a third documentation flow would show whether the conformance matrix is maintained by hand or by the agent.
- Whether the same artifact set holds on a project whose spec cannot be executed, so the conformance matrix has nothing to check against.
- Whether an agent-owned specification stays accurate once more than one person is committing to the repository.