Build1 publisher3 min readPublished
A small labelled holdout turns a miscalibrated chat LLM into a working abstention gate
A study of 14 chat-tuned models found their maximum softmax probabilities overconfident everywhere and uncorrelated with task accuracy, while those same scores still sorted correct answers above wrong ones well enough to drive selective abstention.
The Engineer · Build desk
What happened
- A study of 14 chat-tuned language models found their maximum softmax probabilities consistently miscalibrated on multiple-choice Q&A, and overconfident in the same direction across the set.
- The same scores still ranked answers usefully: wrong answers carried smaller MSPs than correct ones, a result that held under statistical testing for models performing well on the task.
- Across models, Q&A accuracy correlated strongly with how well the MSP predicted correctness, and showed no correlation at all with calibration error.
- Letting a model abstain when its first response fell below an MSP threshold improved performance, with the threshold chosen from only a small amount of labelled data.
- The models spanned open-source and proprietary systems of varying size and architecture, tested on five Q&A datasets under two different question phrasings.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint An error budget written as a fixed probability cutoff is calibrated against nothing, and the authors expect calibration not to improve with capability, so the cutoff will not become meaningful by upgrading the model underneath it.
- cost Abstention on this design needs a labelled holdout per model and prompt to set the threshold, and that labelling cost recurs every time either changes.
- exposure Because correctness prediction tracks accuracy, the confidence gate is weakest on the weak models and unfamiliar domains where an escalation path is worth building.
- capability Teams can get selective abstention without fine-tuning a refusal behaviour into the model, which the authors describe as costly and a risk to the model's other capabilities.
The maximum softmax probability is the largest probability the model puts on any one candidate answer, and the paper treats it as uncertainty information the model already carries [10]. Calibration would mean something exact about that number: among responses scored p, a fraction p are correct [7]. That property is what buys an unsupervised policy. Answer only above a threshold and your error rate is bounded by the complement, with nothing labelled [7]. The study's models are overconfident [1], so the bound does not hold.
Ordering is the part that survives. For models that do well on the underlying Q&A task, wrong answers carry lower MSPs than correct ones, and the authors report this under rigorous statistical testing [2]. Selective abstention on the MSP of the first response improves performance, with the threshold picked from a small amount of labelled data [5].
The sweep is 14 chat-tuned models across five Q&A datasets with two question phrasings [1][6], which is 140 model-dataset-phrasing cells if every combination was run [13]. Two phrasings is the minimum check that a ranking signal is not an artifact of one prompt template. The measured task is multiple-choice Q&A [1]. For the threshold to transfer to your router, your task has to look like that: a closed option set where one softmax maximum stands in for the whole answer. Free-form generation is outside what this paper measured.
Two cross-model correlations decide whether the miscalibration is temporary. Q&A accuracy correlates strongly with MSP correctness prediction, and does not correlate with calibration error at all [3]. The authors' reading is that under current chat fine-tuning methods, teams can expect correctness prediction to improve as capabilities advance and calibration not to [4]. So a hard-coded 0.9 cutoff will still be miscalibrated after the base model underneath it improves.
The same correlation has an uncomfortable consequence for escalation design. Correctness prediction is best where accuracy is already high [3], and the ranking hypothesis held for models that perform well on the task [2]. So the gate is sharpest on queries a strong model would have answered correctly anyway, and least reliable on the weak model or the unfamiliar domain where the escalation was the point. A threshold fitted from labelled data is a fitted parameter [5], tied to the model and the prompt that produced those labels; I would re-fit it on every model version bump.
The approach this replaces is training the behaviour in. Prior work fine-tunes models on labelled data so they recognise questions beyond their knowledge, which the authors describe as costly and as something that could interfere with the model's other capabilities [8]. Their motivating failure is concrete: an LLM-based system invented a set of nonexistent court cases, and two lawyers were formally sanctioned [11]. The question they set instead is whether "this ability [can] be invoked rather than learned" [9].
What to watch
- Whether anyone replicates the ranking result on free-form generation, where no single softmax maximum stands in for the answer.
- Whether a later model generation breaks the reported absence of correlation between Q&A accuracy and calibration error.
- Whether follow-up work reports how stable a fitted MSP threshold is across the two question phrasings the study used.