Evaluating RAG systems

Introduction

A retrieval-augmented generation system is the easiest AI feature to demo and one of the harder ones to keep honest. You point it at a corpus, ask a question the corpus answers plainly, and it answers; whoever is watching nods. The next question, from somebody who actually works with the material, comes back wrong in a way nobody notices until it matters: a fluent paragraph citing a page that says something else, an answer silently missing the one document that mattered, a confident figure assembled from two versions of the same record.

None of that shows up in a demo: a demo is a curated sample with a human choosing the next question, and it measures nothing. There is no failing set behind it, so there is nothing it can fail.

I treat evaluation of a retrieval system as an engineering discipline rather than a vibe check. The unit of work is a set of questions with known right answers, run on every change, behind a threshold that fails the build when quality drops. That is duller than a demo, and the only thing I have found that keeps a retrieval system from drifting away from the corpus it represents.

What a RAG system actually does

Every stage owns a failure, and each failure needs a different measurement.

Ingestion reads the corpus, parses each document into text, splits it into chunks, attaches metadata, and writes vectors plus a searchable representation into an index. Retrieval turns a question into one or more queries and pulls a candidate set back — usually a cheap lexical pass, a vector pass, or both. Ranking reorders that set, sometimes with a cross-encoder, sometimes with rules such as recency or document authority. Context assembly decides what fits in the prompt: how many chunks, at what length, in what order, and what is dropped when the budget runs out. Generation writes the answer from that context and the instruction. Citation is the last stage and the one most often left to chance: the model is asked to point at the chunks it used, and the pointers are either checked or not.

Two or three of those are expensive to get wrong — chunking and the index boundary in particular, because they decide what retrieval can ever find. That is the same kind of call as a service boundary, and worth a second opinion before it is frozen: the point at which I would reach for architecture and technical advisory, rather than discover the cost nine months into a corpus re-ingested four times.

If retrieval calls tools — a document fetcher, a search API, a database lookup — or the loop is agentic and picks its own next query, the surface widens, because the trajectory becomes part of the artefact under test. The operational side of that is in MCP servers in production; here the measurement problem is the same one with more moving parts.

A failure taxonomy worth naming

The first useful thing an evaluation effort produces is vocabulary. “It gave a bad answer” is not actionable; “retrieval missed the only document that contained the figure” is. These are the classes I name, grouped by the stage that causes them.

Index-side failures

A retrieval miss is the plainest one: the chunk that answers the question is not in the candidate set at all. A query that shares no vocabulary with the document, an embedding space that puts the relevant passage nearer an irrelevant one, or a filter that excluded it silently will all do it. Misses are the expensive failure, because no prompt recovers information that never arrived.

A stale index is the same failure on a timer. Documents change, get deleted, get superseded, and the index keeps serving the old text with full confidence; if ingestion is a batch job somebody remembers to run, the index is stale on a schedule nobody can state. Deletions are worse than updates, because a retracted document retrieves beautifully and answers wrongly.

Chunking artefacts live at the seam between ingestion and retrieval. A table split down the middle loses its header row; a procedure split after step three returns the first half as if it were the whole thing; an answer spanning two sections is in neither chunk. Chunk size is a real trade-off: small chunks retrieve precisely and arrive without context, while large chunks carry context and dilute the signal that made them retrievable.

Retrieval and ranking failures

A ranking error is subtler. The right document is in the candidate set, at position forty, behind three near-duplicates and a glossary entry. If the context budget is ten chunks, ranking error looks like a retrieval miss from the outside and needs different work to fix.

Retrieval can also succeed on paper and fail in effect: the correct chunk is retrieved, ranked first, and the answer is still wrong, because the question needed two documents and one fit, or a second chunk contradicted the first and the generator picked the wrong side. Retrieval metrics alone cannot certify a system, which is why the generation side needs its own measurement.

Assembly failures

Truncation and context overflow happen when the budget is spent on material that does not help: headers, navigation text, boilerplate, or a chunk long enough to crowd out the two that mattered. The symptom is an answer technically supported by something in the long context that does not answer the question — the model found a sentence it could use and used it. Order matters too: a chunk buried in the middle of a long context is used less reliably than the same chunk at the top.

Generation failures

Unsupported generation costs the most trust: an answer that reads as authoritative, contains a specific figure, and cannot be traced to any retrieved chunk. It is why citation is a stage rather than a courtesy. A citation pointing at a real chunk that does not support the claim is worse than none, because it converts a plausible answer into a verifiable-looking one.

Refusal behaviour belongs here too. A system that answers questions its corpus cannot support is not being helpful, it is confabulating on demand; one that refuses questions the corpus does support is equally broken and harder to notice, because a refusal reads as prudence. Both directions need cases, including questions deliberately outside the corpus.

None of this is exotic. Every failure here is a boundary problem — between documents, between versions, between what the index holds and what the answer claims — which is why index and data boundaries deserve deliberate design rather than accretion, the argument I made for services in Micro Mono Modu.

Build the failing set first

Every other part of this — metrics, gates, dashboards — is downstream of one artefact: a set of real questions with known correct answers. Build that first. A system with metrics but no failing set is measuring the wrong thing precisely.

Collecting questions without leaking the answer

The source material is the people who use the thing: support tickets, internal chat, the questions analysts ask each other, search queries that returned nothing. The distinction that matters is between a question and an answer. Write the question down exactly as it was asked, abbreviations and typos included, because those are load-bearing — a cleaned-up question evaluates a system nobody uses.

Then record, separately, what a correct answer looks like: which passages support it, which figures it must contain, what it must not claim. That is not tidiness. It is what lets the set be handed to somebody who knows the corpus, and to a judge, without the answer sitting in the question wording. A question phrased as “what is the retention period for the audit records, which is seven years” measures a system’s willingness to agree with its prompt, not its retrieval.

The first version is small. Thirty questions spanning the corpus honestly beat three hundred generated in an afternoon, and they are enough to start, though not enough to certify anything. I would rather ship a small set that gets maintained than a large one that becomes a museum exhibit.

Graded relevance

Binary labels — relevant or not — throw away the most useful information. Grade instead: a passage that fully answers, one that partially answers, one that is on topic and answers nothing, and everything else. Grading costs more per item and pays twice, because it lets ranking metrics tell “the right document was third” from “the right document was fortieth”, and it makes raters’ disagreements visible early, when an ambiguous question can simply be dropped.

The labels are the weakest point in the measurement, and it is worth saying so out loud. If two competent readers of the same corpus disagree about whether a passage answers a question, the metric built on that label is arithmetic on a coin flip. Label noise dominates everything downstream: a retrieval score that moves by two points on a noisy set tells you nothing.

Synthetic queries and their contamination

Generating questions from documents is a reasonable way to get from thirty items to a few hundred, with two caveats that are easy to ignore. Contamination is the first: a question generated from a chunk shares vocabulary with it, so a lexical retriever finds it, and the set sings that retriever’s praises while ignoring the vocabulary mismatch that breaks it on real questions. Mixing in paraphrases — same question, different words, no shared rare terms — is the cheapest correction. The second caveat is that generated questions inherit the document’s framing, so they never ask the question the document fails to answer, which is the one users keep asking.

Keep generated items in a separate, clearly marked slice of the set. Then you can see whether a change improves real questions, and a suspiciously clean synthetic score has its provenance visible in the data.

Retrieval metrics and their caveats

Four numbers cover most of what retrieval needs to say, and each has a shape that invites over-reading.

Recall@k is the fraction of questions with at least one relevant passage in the top k, which answers the question that matters most: could the generator have answered this at all? Precision@k is how much of the top k was relevant, which is really a question about cost: how much of the context budget goes to material nobody needs. Mean reciprocal rank weights how high the first relevant passage sat, a decent proxy for whether the generator will use it. nDCG goes further with the graded labels and discounts by rank, so moving a partially relevant passage up does not score the same as moving a fully relevant one.

Two honest caveats come with all four. They measure retrieval against labels, so a retrieval metric is only as good as the labelling, and label noise propagates straight through. And k is not a free parameter to be tuned for a nicer number: k is a function of the context budget the generator actually has, so if ten chunks fit after the instruction, the question and the answer, then k = 10 is where the metric is taken, and a recall figure at k = 100 describes a system that does not exist. Setting k to whatever makes recall look respectable is the most common way a retrieval evaluation becomes fiction.

Recall@k and nDCG are cheap to compute against the labelled set, and writing them by hand once is worth doing before trusting a framework’s version of them.

from math import log2

RELEVANT = 2  # grades: 0 irrelevant, 1 on topic, 2 partial, 3 full


def recall_at_k(ranked: list[str], graded: dict[str, int], k: int) -> float:
    relevant = {chunk for chunk, grade in graded.items() if grade >= RELEVANT}
    if not relevant:
        return float("nan")
    return len(relevant & set(ranked[:k])) / len(relevant)


def ndcg_at_k(ranked: list[str], graded: dict[str, int], k: int) -> float:
    discount = lambda position: 1.0 / log2(position + 2)
    gains = [graded.get(chunk, 0) for chunk in ranked[:k]]
    ideal = sorted(graded.values(), reverse=True)[:k]
    dcg = sum((2**grade - 1) * discount(i) for i, grade in enumerate(gains))
    idcg = sum((2**grade - 1) * discount(i) for i, grade in enumerate(ideal))
    return dcg / idcg if idcg else float("nan")

Generation and grounding metrics

Retrieval metrics tell you whether the answer was reachable. They say nothing about whether it was correct, and that gap is where most of the value and most of the risk sits.

Correctness and faithfulness

Answer correctness is the blunt one: does the answer agree with the reference answer, on the facts, including the numbers? Grade it on a small ordinal scale rather than as pass or fail, because “right but missing the caveat” and “wrong about the retention period” are not the same size of problem.

Faithfulness is narrower and more useful: is every claim in the answer supported by the retrieved context? A system can be faithful and wrong, if the corpus itself is wrong, and it can be correct and unfaithful, arriving at the right figure through a step the context does not license. For a retrieval system, unfaithfulness erodes trust fastest, because the answer looks checkable and is not.

Citation accuracy

Citation accuracy is a metric in its own right and the one most often skipped. Three questions per answer: does the cited chunk exist, does it support the specific claim attached to it, and does the claim that most needed support carry a citation at all? A citation attached to a chunk that merely mentions the topic scores as present and reads as verified, which is exactly what the metric has to catch. Checking it needs the claim and the chunk side by side, which no string comparison does; it is a judge task or a human task with a rubric, and pretending otherwise is how citation coverage gets reported as citation accuracy.

Refusal behaviour

A model-backed feature has three possible responses, not two: answer, refuse, and answer with a hedge that sounds like an answer. Testing refusal needs questions the corpus genuinely cannot support, including adjacent topics that share vocabulary. The interesting cases are the near misses: a question that sounds like the corpus but is not, and a question the corpus answers only in one obscure chunk. Both are where an unmeasured system is most confident and least reliable.

Judge models and calibration

Judging at any volume means a model, with a rubric, scoring against the reference. The rubric is written before the judge sees the answers, and it is specific enough that a person would score the same way: what counts as a required fact, what counts as an unsupported claim, whether a missing caveat is a one or a zero. Rubric writing is the actual work; the judge is the part that runs at scale.

Then calibrate. Take a sample that a person has scored and check whether the judge agrees, and look at where it disagrees rather than only how often: a judge that gives partial credit to padding, or that rewards a confident tone, is worse than useless, because it approves exactly the answers a reader would reject. When the rubric changes, recalibrate. A judge score is a measurement taken with an instrument, and instruments drift.

Judge cost and drift

Budget the judging. Judge calls cost money and time, so how many items get judged per run is a design decision like any other: a large judged set nightly, a small high-value subset on every commit. The judge also has its own version — model, prompt, rubric, temperature — and a silent upgrade under a pinned name will move scores with no corresponding change in your system. Record the judge’s identity in every result, treat a judge change as a change worth its own run, and keep the frozen human-scored sample as the anchor that shows how far the instrument has moved.

The harness in CI

Metrics that live in a notebook are a report. Metrics that run on every change and can fail the build are a control, and the difference between the two is most of the value.

The harness has the same shape as the system it measures. A pinned corpus snapshot and a versioned query set go in; retrieval, context assembly and generation run exactly as they do in production; the outputs go to graded judges and to a sampled human review; the results are compared against stored thresholds, and the gate decides whether the change goes out. Nothing in the run is allowed to differ from the deployed system, or it is measuring a system that does not exist.

flowchart TD
  SNAP["Corpus snapshot<br/>pinned, digest recorded"] --> PARSE["Deterministic parse and chunk"]
  PARSE --> INDEX["Index build"]
  SET["Query set<br/>dev and held-out test"] --> RET["Retrieval at the k the budget allows"]
  INDEX --> RET
  RET --> ASM["Context assembly"]
  ASM --> GEN["Generation"]
  GEN --> JUDGE["Graded judges: correctness, faithfulness, citation"]
  GEN --> HUMAN["Sampled human review"]
  JUDGE --> STORE["Results keyed by snapshot, config, judge version"]
  HUMAN --> STORE
  STORE --> GATE{"Regression gate: thresholds plus cost and latency budgets"}
  GATE -->|"pass"| SHIP["Deploy"]
  GATE -->|"fail"| STOP["Fail the build, report the per-query diff"]

A run is a reproducible artefact

A score with no provenance is an opinion with a decimal point. Every run records the corpus snapshot it was taken against, the parser and chunker version, the embedding and generation model versions, the ranking configuration, the judge’s identity and the query set version.

Determinism is the part worth engineering. Fetch the corpus from a pinned snapshot rather than a live source, or a run at three in the morning measures a different corpus than the run that set the baseline. Parse the same bytes the same way, with the parser output cached and digest-checked, because a parser upgrade that shifts chunk boundaries is a real change and should show up as one. Pin the judge, and treat a judge change as its own run.

Gates, thresholds and budgets

Thresholds belong in version control next to the code, not in a slide. Each one is a floor and a tolerance: a hard floor on the metrics that describe correctness, a band on the ones that fluctuate, and an explicit statement of which failures may be reported rather than block.

Quality is not the only budget in the gate. Per-query cost and latency belong there too, since a retrieval change that lifts recall by adding a reranker and four hundred milliseconds is a trade, not a win, and the gate is the only place the trade is visible before users feel it. That is where AI in the delivery pipeline earns its keep — generating candidate questions, proposing rubric criteria, expanding tests around untested paths — and where the guardrails that stop it becoming a rubber stamp matter: every new case is reviewed by a person before it joins the set, and nothing joins the gate that somebody would not defend in review. That is the practice behind the AI-augmented engineering work.

Two sets, not one, is the last structural piece: one that you develop against and are allowed to look at while you tune, and one held out that you look at rarely. When a change improves the dev set and does nothing on the held-out set, the change fitted the set. Without the split, “improving the evaluation score” and “improving the system” become the same sentence, and the gate stops being a control.

# evals/config.yaml — thresholds live with the code they gate.
corpus:
  snapshot: s3://corpus/2026-10-26 # pinned; digest recorded per run
  parser: pdf-ingest@3.4.1
retrieval:
  k: 10 # the context budget, not a tuning knob
  rerank: cross-encoder@1
generation:
  model: chat@2026-09
  citations: required
judges:
  rubric: rubrics/grounding.v4.yaml
  model: judge@2026-09
  sample_rate: 0.05 # sampled for human review
thresholds:
  recall_at_10: { floor: 0.90, tolerance: 0.02 }
  ndcg_at_10: { floor: 0.78, tolerance: 0.03 }
  citation_support: { floor: 0.95, tolerance: 0.01 }
budgets:
  p95_latency_ms: 2500
  cost_per_query: 0.004
sets:
  dev: evals/sets/dev-2026-10.jsonl
  test: evals/sets/holdout-2026-10.jsonl

Results worth diffing

Store one record per query, not one number per run. A score is a summary of a diff you may need to read the next morning, and the useful question is always which queries changed.

{
  "run": "2026-10-26T09:14:02Z",
  "snapshot": "s3://corpus/2026-10-26#sha256:0f5c…",
  "config": "evals/config.yaml@a41c9e2",
  "query": "how long are 2019 audit records retained",
  "retrieved": [
    { "chunk": "pol-retention-v3#p12", "rank": 1, "grade": 3 },
    { "chunk": "pol-audit-scope#p2", "rank": 2, "grade": 1 }
  ],
  "generated": {
    "answer": "Seven years from the end of the financial year.",
    "citations": [{ "chunk": "pol-retention-v3#p12", "supports": true }],
    "refused": false
  },
  "judged": { "correctness": 3, "faithfulness": 1.0, "citation_support": 1.0 },
  "human": null,
  "cost_usd": 0.0031,
  "latency_ms": 1840
}

When evals lie

An evaluation harness is a measurement you built yourself, against a set you also built, and it will happily report that a worse system is better. The failure modes are consistent enough to check for by name.

Two deserve a sentence more. Contamination is not only synthetic queries: a model trained on the open web has probably read the documents, and possibly a public version of the very questions you are asking, which is a reason to keep part of the set private and to write questions in your users’ words rather than the corpus’s. Lenient judges are the quiet one, because a judge that agrees with everything produces a green build and no argument.

The general sanity check is to break the system on purpose. Delete the top-ranked chunk and see whether the answer metric notices. Swap two chunks, corrupt one figure, ask a question whose answer is on page forty of a document nobody retrieved. A harness that reports the same score after sabotage is measuring the harness.

Production feedback

A frozen set ages. The corpus changes, the users change, and the questions that hurt are the ones nobody thought to write down in October.

Sampled human review is the anchor: a small, regular sample of real answers, read by somebody who knows the material and can say whether the citation supported the claim. It is slow, it does not scale, and it is the only source of truth about whether the judge is still calibrated. Two things make it usable. The sample is drawn at random rather than from complaints, and the reviewer records a verdict against the same rubric the judge uses, so the two numbers are comparable.

Implicit signals — a follow-up question, a copied answer, a session abandoned — are cheap, tempting, and weak evidence dressed as strong evidence. A user who gives up tells you something; a user who accepts a wrong answer tells you nothing. Before logging any of it, decide what question text you are allowed to keep, redact identifiers at ingestion, and treat the review queue as personal data with a retention period, because it usually is.

The loop closes when a wrong answer becomes a case. Copy the question verbatim, write down what a correct answer is and which passages support it, grade the passages, and put it in the dev set that week. A defect that is not in the set will be reintroduced by the next prompt change.

Cadence and ownership keep it alive: a weekly sample read by a named person, a monthly review of the set for duplicates and dead questions, and a rule that whoever changes retrieval owns the resulting diff. A set with no owner becomes the museum exhibit described earlier.

What to measure before you tune

1. Thirty real questions, written down verbatim, each with a reference answer.
2. Graded relevance for the passages that support each answer.
3. Recall@k at the k your context budget actually allows, and nDCG@k.
4. Faithfulness and citation support, judged against a rubric, calibrated on a human-scored sample.
5. A pinned corpus snapshot, parser and judge, recorded per run.
6. Floors on the metrics that describe correctness, plus per-query cost and latency budgets.
7. A dev set you can look at and a held-out set you cannot.

The temptation to avoid is the one number. A composite score that rolls retrieval, faithfulness, citation and cost into a single figure is easy to put on a dashboard and useless when it moves, because it cannot say which stage failed. Report the vector, per stage, and let a person read the diff.

Closing

Retrieval systems fail in the seam between what the corpus says and what the answer claims, and the only way I have found to keep that seam honest is to make failure routine: a set of real questions, a harness that runs on every change, and a gate that can stop a deploy. It is unglamorous work, and it is the difference between a feature that can be trusted in use and a demo that passed its own review.

The same instinct runs through the AI-augmented engineering work: AI is worth having in the pipeline when it is measured by what it saves and reviewed like any other change, with a person accountable for the result. And for a model-backed feature, a passing gate is part of what done means now, alongside the tests and the runbook. If that is the part you are missing, it is usually a short piece of work to build it.