SciCon subtracts the text-only score at decode time to stop figure QA from guessing

A preprint posted to arxiv.org proposes SciCon, a training-free decoding rule for scientific figure multiple-choice question answering that scores each candidate answer by subtracting a text-only option score from the same option's image-conditioned score ^[1]. The authors report that it improves accuracy over standard decoding baselines across three scientific figure QA benchmarks and three model backbones ^[2], which makes it an inference-time patch rather than a data or fine-tuning project ^[1].

The failure mode is worth naming precisely, because it is easy to mistake for a vision problem. In scientific MCQA, the answer choices are not neutral alternatives to rank; they carry strong semantic and domain-specific cues that function as priors on their own ^[3]. A model can therefore prefer an option because it reads as scientifically plausible in text, even when the figure supports a different answer ^[4]. The authors call this choice-induced prior bias, and its decode-time expression text-prior-dominant decoding: the final prediction stays aligned with what the model would have said from the question and choices alone ^[5].

That framing is what separates SciCon from the existing contrastive decoding literature. Earlier methods reduce hallucination by contrasting the original input against a distorted image or a perturbed instruction; SciCon instead targets the prior encoded in the candidate text itself ^[6]. The background is not new: prior work has shown vision-language models over-relying on language priors and memorized associations instead of genuine visual reasoning ^[7], and recent benchmarks show that even advanced multimodal models struggle with figure reasoning, which requires reading trends, comparing panels, and mapping symbolic abstractions onto domain meaning ^[8]. Most vision-language progress has been measured on object-centric, general-domain tasks such as recognition and captioning ^[9]. The stakes are practical: figures often carry a paper's central empirical claims in visual form ^[10], and agentic systems increasingly depend on grounding in scientific documents and their visual evidence ^[11].

Two engineering consequences follow from the arithmetic. Because every candidate needs both an image-conditioned score and a text-only score, the scoring work per question is roughly double that of standard decoding ^[12]. And the recipe assumes your stack can read per-candidate scores; a pipeline that only receives the model's chosen letter has nothing to subtract from ^[13]. Neither cost is exotic, but both land in the serving path rather than the training budget.

The abstract claims consistency, not magnitude: it does not state how large the gains are, which benchmarks or backbones were used, or whether the subtraction is weighted ^[14]. Watch for those numbers in the full paper, and specifically for the regression case, since questions whose plausible-sounding option is also the correct one are exactly where a text-prior penalty should hurt. The authors say code is available at github.com/dmis-lab/SciCON ^[15], which is the fastest way to check whether the subtraction carries a tuning coefficient and how it behaves when the figure is genuinely uninformative.