Reading faithfulness against context recall tells you which half of a RAG pipeline broke

Faithfulness is a ratio, and the judge sets its denominator. The answer is split into individual statements, and each statement is checked against the retrieved chunks ^[6]. Whatever number of statements the judge finds is the number the score divides by ^[21]. Split one long sentence into three statements and the fraction changes while nothing in the pipeline moved.

The division between retrieval and generation does not hold for answer relevancy. Its target is 0.75 or higher, and it asks only whether the answer addresses the question that was asked, not whether the answer is correct ^[10]. The dev.to article describes high faithfulness with low answer relevancy as a retrieval problem that looks like a generation problem: the answer was derived correctly from the context, and the context was the wrong context ^[11]. Two other pairings do point straight at one half. High faithfulness with low context precision or recall puts the fault in retrieval ^[14]. Low faithfulness with good context precision and recall means the retriever found the documents and the generator ignored them ^[15].

Context recall catches the documents that never arrived at all. The article lists the usual causes as chunks that are too small, an embedding model that does not map domain-specific terms, or a top-K set too low ^[13]. Its target band is 0.75 to 0.8 ^[13].

For the division of labour the article cites a study in the Journal of Machine Learning Research. Retrieval accuracy explains about 60 percent of the variance in final answer quality, and the remaining 40 percent is attributed to how the model uses the context it was handed ^[3]. If that ratio holds on a given corpus, scoring the retriever alone leaves the other 40 percent unmeasured. BLEU and ROUGE score surface text similarity, which the article notes has nothing to do with factual correctness or contextual fidelity ^[4].

The threshold numbers are the part I would not adopt as printed. They are given as recommendations and target values, with no corpus named ^[22]. For the production threshold to mean pass on your system, your test questions have to decompose into statements at roughly the same granularity as the ones behind the recommendation, and your judge has to be about as strict.

Then the bill. At the top of the quoted per-case range, a 200-case suite costs 6 dollars a run, and ten CI runs a day is 60 dollars a day ^[20]. DeepEval exposes the metrics as pytest-compatible tests ^[17], and per-commit evaluation is what turns a per-case figure into a monthly line. The suite that catches the model making things up is itself made of model calls, and each one is billed ^[18]. For production monitoring the article points to Patronus for hallucination detection and Langfuse for tracing ^[17].