Build1 publisher2 min readPublished
A SHA-256 run manifest shows which conditions changed between two eval attempts
Ranknod's dev.to guide digests eight eval conditions, including the resolved model and the rendered messages, into one SHA-256 manifest ID. Any change to a recorded input changes the ID, though the post says a match cannot promise identical output.
The Engineer · Build desk
What happened
- Ranknod's dev.to post starts from an eval that improved after a prompt edit and could not be reproduced a week later, even though the prompt file was still on disk.
- It says to record the resolved model ID, endpoint configuration, generation settings, rendered messages, ordered context, tool definitions, adapter revision and rubric revision.
- An accompanying Python 3.12 script reduces that record to a single SHA-256 configuration ID.
- Its fixture uses synthetic model names, and the author says they make no claim about how any provider versions its models.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- capability A reviewer who sees matching IDs on two runs can rule out recorded configuration as the cause before crediting a prompt edit with the improvement.
- decision Teams have to keep two identities per attempt, a configuration digest that retries share and a run ID holding timestamps, outputs and durations, or every retry looks like a config change.
- cost Large inputs have to be kept as immutable references or protected artifacts, so adopting the manifest adds an artifact store that a folder of prompt files never needed.
The script in Ranknod's dev.to post is four short functions [14][6]. `validate_json_value` walks the structure and checks exact types with `type()` [8]. Anything outside the allowed set raises `TypeError` with the message "use string keys and JSON values; encode decimals as strings" [8]. `stable_bytes` calls `json.dumps` with `sort_keys=True`, `ensure_ascii=False` and `separators=(",", ":")`, then encodes to UTF-8 [6]. `manifest_id` hashes those bytes with SHA-256 [6].
Floats are refused on purpose. The schema accepts strings, integers, booleans, nulls, lists and string-keyed dictionaries, and the author says the limit keeps the representation easier to explain [7]. So the fixture stores temperature as the string "0" beside an integer `max_output_tokens` of 300 [10]. I would make the same call. Once a decimal is a string, the hash covers exactly the characters someone typed.
The fixture checks two cases [9]. One copy of the manifest has its keys in reversed order. Another bumps the context note from revision v1 to v2 [9]. Because keys are sorted before hashing, the reordered copy gets the same ID and the bumped copy gets a different one [2].
Large inputs enter as nested digests. `messages_sha256` fingerprints the rendered messages, `tools_sha256` covers the tool list, and each context entry keeps its position, ID, revision and a hash of the source bytes [11]. All of that depends on the adapter. The post says a real adapter "must supply the actual artifacts used by the request" [11]. An adapter that hashes the template file instead has computed a more expensive filename. Behind a stable template name, variables may have been substituted, earlier messages retained, retrieved passages reordered or a tool schema changed [1].
The model field has the same limit. The post asks for the resolved model identifier "where the provider exposes one" [3]. Where the provider does not, the manifest can only hold the alias. A repointed alias then changes no recorded field, and the ID stays the same [3]. An alias that may have moved is one of the causes in the post's opening failure [2]. The author scopes the claim to match: the digest "helps detect changes to the recorded configuration; it does not guarantee identical model output" [4].
What to watch
- Whether model providers return a resolved model identifier on every response; where they do not, the manifest can only record an alias that can move.
- Whether the author publishes a run of the scheme against a real provider instead of synthetic model names.