Skip to content

Build1 publisher3 min readPublished

Rounding a judge model's 0.99 into a price tier billed a synthesis as a lookup

A dev.to developer prices each API request after the work by asking a second model how much the answer represents, and the first version threw away the calibration he was paying for by rounding its probabilities into three tiers.

The Engineer · Build desk

Photograph accompanying Rounding a judge model's 0.99 into a price tier billed a synthesis as a lookup
Photo: dev.to

What happened

  • The rewrite charges the probability-weighted mean of the level prices, so 12% synthesis and 88% investigation at $0.025 and $0.05 settles at $0.047.
  • A second question about whether answers were grounded scored between 0.07 and 0.54 on every call until the sources were put into the state, after which grounded answers scored 0.98.
  • A Vercel AI Gateway key sent to TypeSafe's own endpoint returned 401, and the rule-based fallback kept pricing requests while the rejection sat in a reason string nobody was reading.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • exposure A fallback in the billing path can price at rule rates for as long as the judge keeps failing, and the buyer only learns which one set the price if the receipt names it.
  • decision Converting a grounding probability into money is a policy choice the operator has to make and publish: multiply the price by it, or refuse to bill below 0.5.
  • constraint Keeping the pricer separate from the producer is harder in detection than in text, because SAM 3 both names the instances and scores them.

The settle-less-later half of the payment flow was already available: authorize a ceiling before the work starts, settle a smaller amount after [2]. The payment layer leaves how much smaller up to you, and the number has to arrive fast enough to sit inside the request. It also cannot come from the model that produced the work [3]. Jev answers typed questions with probabilities and a confidence, and returns no prose [4]. The first version asked it one question, how much work the answer represents on a three-level scale, and mapped the reply to a tier [5]. On a scale whose levels are lookup, synthesis and investigation, a 0.99 means essentially a synthesis. The code floored it to 0 and charged for a lookup [6]. "The distribution is the product. The moment you round it into a bucket you have thrown away the only thing that distinguishes a calibrated model from a general one asked for JSON," the author wrote [7]. The replacement is a single reduce over the levels, multiplying each level's price in micros by its probability [8]. At 12% synthesis and 88% investigation, with those levels priced at $0.025 and $0.05, the settled amount is $0.047 [9]. Rounding to the likelier level would charge $0.05, three tenths of a cent high; flooring to synthesis would charge $0.025, 2.2 cents low, an error roughly seven times larger [10]. The post does not give a price for the lookup level, so the cash size of the original 0.99 undercharge cannot be computed from it [11]. Buyers do not get a standing discount out of that undercharge. Rounding makes the price jump at an arbitrary line, and whether the jump favours the seller or the buyer depends on where the line sits [12]. The mean is unbiased in a narrow sense: if the probabilities are calibrated, the charge converges on the expected value of "the value function I chose, not some true worth of the work" [13]. Charging a mean needs somewhere to hold it. The ledger is denominated in micros, so $0.0235 is the amount that settles and not a display rounding [14]. A judge split between two levels then charges between them, with no hand-tuned rule for uncertainty [15]. The second question asks whether the answer is grounded in what the service actually read. Every call came back between 0.07 and 0.54, and the author spent a while believing the model was bad at the task [16]. He was sending the question and the answer and asking whether it was supported by the sources, without sending the sources; with them in the state, grounded answers score 0.98 and ungrounded ones stay low [17]. "A verification question with nothing to verify against is a vibe check, and it will happily return numbers that look like data," he wrote [18]. He multiplies the expected price by the grounding probability, pricing a wholly ungrounded answer at nothing and a half-grounded one at half, and says another implementer could floor it or refuse to bill below 0.5 [19]. "Whatever you choose, put it in the receipt," he wrote [20]. Every judge path falls back to rule-based pricing on a missing key, a timeout, an error or an unrecognised shape [21]. So the 401 stayed invisible. With a real key set, prices kept coming out of the rules, because the key was a Vercel AI Gateway key sent to TypeSafe's own endpoint and the fallback swallowed the rejection into a reason string nobody was reading [22]. Responses now state which judge priced the call and why [23]. Text is the easy case: detection only gets a price once it runs, and the model that answers also scores its own answers [24].

What to watch

  • A published calibration check on Jev's work-level question, since the expected-value price only converges if those probabilities are calibrated.
  • Whether the detection path gets a judge independent of SAM 3's own instance scores, or prices off the producer's confidence.
  • Whether the judge-and-reason field in the receipt becomes something a buyer can dispute a charge with.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories