Skip to content

Build1 publisher3 min readPublished

ADR-0001 demotes Langfuse to a projection of Kept's in-process trace

Kept's tracing layer went in before the agent loop, which produced one design decision worth copying and a promised accounting of the drawbacks that the available text never reaches.

The Engineer · Build desk

Photograph accompanying ADR-0001 demotes Langfuse to a projection of Kept's in-process trace
Photo: dev.to

What happened

  • The developer of Kept, a self-hostable post-purchase support agent for e-commerce, designed and implemented the entire tracing layer before writing any code for the main agent loop.
  • A research phase surfaced two trace consumers with incompatible needs: humans analyzing, searching and filtering in a UI, and evals needing fast access to a reliable record of what happened.
  • A span left in_progress when its trace ends is recorded as undetermined rather than error, so a process that dies mid-span produces a span with an unknown result instead of nothing.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • cost Any eval run that fetches its trace from Langfuse pays API calls and latency for data the agent process already held, and the concurrency handling comes out of the same engineering budget.
  • constraint With both SDKs barred from core and a script watching the import graph, a later push to emit OTel spans from inside the agent loop has to land at the adapter boundary or trip the check.
  • capability Because each trace carries the faults injected on that run, a fault-injection matrix can be read as attribution rather than as a list of failures needing manual bisection.
  • decision Anyone wiring evals into an agent now has to pick which consumer owns the record, because the human UI and the eval harness want different latency and different guarantees from the same data.

Consider the crash halfway through a tool call. In Kept, ending a trace sweeps every span still marked `in_progress` and stamps it `undetermined` rather than `error`, at `trace.ts#L82-L91` and `span.ts#L92-L105`, so the crash arrives as a span with an unknown result instead of a gap in the tree [17]. That distinction is what the type sketches were built to satisfy. Each round was driven by one self-test: can the types express a model call inside turn 2 followed by a tool span whose result state was unknown [10]. The answer required dropping "every model call is a span" in favour of three span kinds with a parent link, the turn span sitting above model calls and tool executions as siblings, with causality carried by a `callId` [11].

The stated rationale behind the ordering is that logs and metrics matter more for reliability than an ideal architecture or an industry-leading framework [3]. The research phase earned it. The original plan had Langfuse holding all spans and traces and serving them wherever they were needed [5]. Two readers broke that: the human wants a structured UI for analysis, search and filtering, while the evals want fast access to a reliable record, and routing the evals through Langfuse would have put API calls and latency in the eval path along with concurrency cases to account for [5][6].

So ADR-0001 casts Langfuse as a sink for human readers, leaving the machine-facing record to the in-process trace [7], and the in-process trace model became the source of truth that evals read, with Langfuse as a projection of it [8]. The enforcement is the part worth copying: the core package carries neither the Langfuse SDK nor the OTel SDK, and `check-eval-boundary.mjs` holds the line [9]. An ADR that only humans read is a suggestion.

Stamps were picked by asking who reads each one later, guided by the OTel GenAI semantic conventions, and defined at `types.ts#L129-L144` [12]. `promptHash` hashes the template rather than the rendered prompt, so the value means prompt version, which is the grouping an eval can actually compare across runs [13]. `backendKind` exists to localize a failure to adapter or core, and `traceId` stays distinct from `sessionId` so evals can replay a conversation without merging traces into one garbled tree [14][15]. The `faultToggles` array records which faults were injected on that run, so a red cell in the planned CI matrix points back at the toggle that caused it [16].

For the shape to transfer, the evals have to run where the trace object lives. In-process truth is only cheap if the reader is in the process; an eval service that reads history after the fact cannot reach that object and is back to querying a store. The ADR relocates that cost; it does not remove it.

The price of the ordering is not in evidence. The post promises three benefits and two drawbacks, plus advice for applying the approach without a fresh codebase [18], and the available text stops mid-sentence on why `undetermined` is deliberately not `error` [19], leaving all five of those items unpublished [20]. What can be checked here is a design and its rationale. Whether front-loading the tracing layer costs more than it saved is a claim the author has flagged and not yet made.

What to watch

  • The two drawbacks the post promises but has not yet published, and whether either is schema churn once the agent loop lands on top of the trace types.
  • Whether the check-eval-boundary.mjs rule survives the first feature that wants OTel spans emitted from inside core.
  • Whether the CI fault matrix that faultToggles was stamped for can still attribute a red cell when more than one fault is injected.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories