Half the Citations Failed: What a Political Chatbot Audit Says About Your RAG Stack

Just Facts put 100 questions on politically contested topics to the paid versions of ChatGPT, Gemini, Grok and Claude, and Fox News reported the results ^[1]. The coverage went to the ideological split in the scores, but the line that should concern anyone shipping a citation-backed feature is this: roughly half of the sources the chatbots supplied were illegitimate, meaning they either did not support the claim attached to them or did not exist at all ^[3].

The answer-level scores, per Fox News' account, ran as follows. ChatGPT answered 94% of the prompts designed to elicit right-coded falsehoods correctly and 75% of the left-coded ones ^[4]. Gemini scored 91% and 76% ^[5], Claude 91% and 81% ^[6], and Grok inverted the pattern at 73% right-coded and 84% left-coded ^[7]. That is a 19-point spread for ChatGPT, 15 for Gemini, 10 for Claude, and 11 points in the other direction for Grok ^[1]. The question set covered immigration, abortion, climate change, elections, crime, gun control and COVID-19, and was written specifically to draw out falsehoods associated with both the left and the right ^[2].

Now put the two findings side by side. The worst single answer-accuracy figure in the set is 73%, an error rate of 27% ^[7]. The citation failure rate is around 50%, roughly 1.9 times the worst answer-level error rate in the same study ^[2]. In other words, the evidence layer failed considerably more often than the answers did. For a product surface, that inverts the usual assumption that citations are the cheap safety net under a probabilistic answer. A fluent answer with a citation can still fail verification if the cited page is irrelevant, unavailable or fabricated ^[8], and an unsupported citation makes a wrong answer look more credible to the user, not less ^[9].

The caveats are real and they cut in a specific direction. This was a deliberately adversarial political prompt set, not a general accuracy benchmark ^[10], and results of this kind move with prompt wording, question selection, model version, system instructions, retrieval settings, and how the study defines a falsehood and a valid citation ^[11]. Jim Agresti, president and co-founder of Just Facts, told Fox News the exercise was meant to examine misinformation rather than political bias alone ^[12]. None of that makes the citation number safe to ignore, because a broken citation is a mechanical failure rather than a contested judgment call: either the URL resolves and entails the claim, or it does not.

The practical response is to stop scoring these systems with one number. Comparable evaluations measure claim accuracy, citation entailment, source existence and susceptibility to user pressure separately ^[13], which is what distinguishes a model that gets an answer wrong from one that compounds it with invented evidence ^[9]. If you are building on top of these APIs, reproduce the test against your own model versions, retrieval pipeline and approval workflow rather than treating one adversarial audit as a safety assessment ^[14].

Watch for a breakdown of the two failure modes, which the available account does not provide ^[15]: nonexistent sources point at generation, irrelevant ones point at retrieval and ranking, and they need different fixes. Also worth watching is whether any vendor publishes link-resolution and entailment rates for its own cited output.