RAG evaluation in Data Science interviews

Why RAG eval is its own discipline

When an interviewer at OpenAI, Anthropic, or any Series B startup shipping an LLM product asks "how would you evaluate this RAG system," they want to hear that you understand RAG has two failure surfaces stacked on each other — retrieval and generation — and that end-to-end accuracy alone hides which half is broken. Candidates who skip the decomposition get filtered.

A RAG pipeline retrieves top-K documents from a vector store, stuffs them into a prompt, and asks an LLM for a grounded answer. It can fail at retrieval (right docs never enter the context window), at generation (model hallucinates beyond context), or at the interaction (right docs retrieved but ranked too low to attend to). Each failure mode needs its own metric and remediation playbook.

Junior candidates name one or two metrics and stop. Senior candidates name a layered eval stack — component metrics, end-to-end metrics, and human or LLM-judged quality — and explain when each one is actionable. This post gives you that stack with the vocabulary recruiters expect.

Retrieval metrics

Retrieval is an information retrieval problem with decades of established metrics. The interviewer wants to see that you know them and pick the right one for the use case.

Recall@K is the share of relevant documents that appear in the top-K retrieved results. If a query has 3 relevant docs in the corpus and 2 of them land in top-10, recall@10 is 0.67. This is usually the most important retrieval metric for RAG because the generator can only use what made it into the context window. If recall@K is low, the generator is doomed regardless of prompt quality.

Precision@K measures how many of the top-K are actually relevant. High precision matters when context windows are tight or when irrelevant chunks distract the model. A retriever with precision@5 = 0.4 is sending the LLM more noise than signal.

MRR (Mean Reciprocal Rank) captures the position of the first relevant document, averaged across queries. MRR is the metric to watch when only the top-1 result matters — for example, a question-answering bot that quotes a single source.

NDCG (Normalized Discounted Cumulative Gain) uses graded relevance (e.g., 0/1/2/3 instead of binary), discounts lower positions logarithmically, and normalizes against the ideal ranking. Use NDCG when relevance is not binary — for code search, partial matches are common.

All four require labeled data: a set of queries paired with the IDs of relevant documents. Without labels, retrieval evaluation collapses into vibes-based ranking comparisons that do not survive contact with a senior reviewer.

Generation metrics

Once the right context is retrieved, the generator can still hallucinate, refuse, or answer the wrong question. Three metrics dominate the literature.

Faithfulness (sometimes called groundedness) asks whether every factual claim in the answer is supported by the retrieved context. An answer that invents a number not present in any retrieved chunk has faithfulness = 0 for that claim. This is the single most important LLM-product metric for regulated domains — legal, medical, financial.

Answer relevancy measures whether the answer actually addresses the question. A perfectly faithful answer that responds to a different question scores low. This catches the failure mode where the model dodges hard queries by summarizing the context instead of answering.

Coherence covers grammar and logical flow. It rarely fails in frontier models but matters when fine-tuning small open-weight models for cost.

These are typically scored by an LLM-as-judge — a stronger model (GPT-5, Claude Opus, Gemini Ultra-class) prompted to rate the candidate output on a 1-5 scale with a structured rubric.

You are an evaluator. Given the context and the answer, rate
faithfulness on a 1-5 scale where:
5 = every claim is directly supported by the context
3 = some claims are supported, others are extrapolation
1 = the answer contradicts or ignores the context

Context: {retrieved_chunks}
Question: {user_query}
Answer: {model_output}

Return JSON: {"score": int, "reasoning": str}

The judge prompt itself becomes a versioned artifact — change the rubric, change the scores. Treat judge prompts like code.

End-to-end metrics

Component metrics tell you where the system breaks; end-to-end metrics tell you whether it ships.

Correctness compares the model's answer against a curated ground-truth answer. Exact-match works for short factual answers; for longer responses use a semantic similarity score or another LLM-judge prompt asking whether the model's answer is semantically equivalent to the reference.

Helpfulness is the user-perceived dimension — collected via thumbs-up/down ratings in product, or simulated with an LLM-judge for offline eval. Helpfulness can be high even when correctness is low (the answer is wrong but plausibly framed), so always pair the two.

Latency is wall-clock time from query to final token. Production RAG systems target p95 under 3 seconds for synchronous chat, under 30 seconds for agentic flows. Track retrieval latency, time-to-first-token, and total generation time separately.

Cost is the per-query spend on embedding calls, vector store reads, and LLM tokens. A RAG system can be technically excellent at $0.40 per query and economically dead at scale. Senior interviewers push on cost.

RAGAS framework in practice

RAGAS is the de facto open-source library for RAG evaluation. It bundles the metrics above into a consistent API and uses LLM-as-judge under the hood for the qualitative ones.

from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)
from datasets import Dataset

# Each row: question, retrieved contexts, model answer, ground-truth answer
data = {
    "question": [...],
    "contexts": [[...], [...]],
    "answer": [...],
    "ground_truth": [...],
}
dataset = Dataset.from_dict(data)

result = evaluate(
    dataset,
    metrics=[
        faithfulness,
        answer_relevancy,
        context_precision,
        context_recall,
    ],
)
print(result)

Two RAGAS metrics worth calling out specifically. Context precision measures whether the most relevant chunks are ranked highest in the retrieved set — it penalizes the case where the right doc is in position 8 instead of position 1. Context recall uses the ground-truth answer to check whether every claim in the reference is attributable to a retrieved chunk; it is your sanity check on whether the retriever is even capable of supporting the answer.

RAGAS is not the only option — TruLens, DeepEval, and Arize Phoenix all cover similar ground — but RAGAS is the most commonly named in interviews and the easiest to demo on a laptop.

Building a golden set

Every metric above collapses without a curated test set of question-answer pairs with known correct retrievals. Without it, evaluation is opinion.

Size. Aim for 100-1,000 examples for regression testing on each release. Below 100, statistical noise dominates and tiny prompt tweaks look like wins. Above 1,000, you start paying real money for judge calls on every CI run.

Coverage. Stratify across query types — factoid, multi-hop, ambiguous, adversarial — and across corpus domains. A 200-example set with 40 examples per query type gives you separable signal when a regression hits one slice but not others.

Construction. Mine real user queries from logs, cluster them, sample within clusters, and have a domain expert write the ground-truth answer and tag supporting documents. Synthetic LLM generation can bootstrap the set but needs human review before it becomes source of truth.

Versioning. Treat the golden set as code. Pin it to a git commit, tag eval-set releases, and never quietly mutate it — otherwise you cannot compare yesterday's eval to today's.

The golden set turns RAG evaluation from a research exercise into a CI gate. Without one you cannot say this prompt change improved faithfulness by 4 points with a straight face.

Common pitfalls

The most common interview-failing mistake is conflating retrieval and generation failures. A candidate sees a wrong answer, blames the model, and proposes prompt engineering, when in fact the right document was never retrieved. The fix is the layered eval stack — check retrieval recall before touching the generation prompt.

A second trap is using the production LLM as its own judge. If GPT-5 generates answers and also grades them on faithfulness, you bake in the same biases on both sides — the model rates its own hallucinations as plausible because they sound like its own writing. Use a different model family as the judge.

The third pitfall is evaluating on synthetic queries only. LLM-generated test questions are systematically easier than real ones — they avoid typos, ambiguity, multi-hop reasoning, and the edge cases that brought users to your product. A pipeline scoring faithfulness 0.95 synthetically can still ship a 0.7 in production. Mine real logs.

Fourth, teams optimize the wrong layer because they only watch end-to-end correctness. A 3-point gain might come from chunking, retriever, reranker, or prompt — you cannot tell which without component metrics. Instrument the layers.

Finally, ignoring cost and latency until the demo is over is a classic. A RAG system hitting 0.92 faithfulness with a reranker, query rewriter, and three sequential LLM calls per query is technically beautiful and economically unshippable. Bring up cost and latency unprompted in interviews — it is a senior signal.

Related reading

• Hallucinations and LLM evals — Data Science interview
• Embeddings — Data Science interview
• Embedding quality evaluation — DS interview
• Long-context LLM — Data Science interview
• Prompt engineering — Data Science interview

If you want to drill RAG, LLM, and ML-system-design questions like this every day, NAILDD ships 1,500+ interview problems across exactly this surface area.

FAQ

How is RAG evaluation different from standard LLM evaluation?

Standard LLM eval measures the model in isolation — does it answer correctly given the prompt? RAG eval adds a retrieval layer that decides what context the model sees, creating a second failure surface. You need IR metrics like recall@K and precision@K on top of generation metrics, plus context precision to catch retrieval-generation coupling failures. Treating RAG like a vanilla LLM is how teams ship brilliant prompts on top of broken retrievers.

Can I evaluate RAG without ground-truth answers?

Partially. Faithfulness and answer relevancy can be scored by an LLM judge from the question, context, and answer alone — no reference needed. But correctness, context recall, and regression tracking require ground truth. The practical compromise is to ship faithfulness-and-relevancy monitoring in production from day one, while building a golden set of a few hundred labeled examples for offline regression testing.

How often should I run the full eval suite?

On every prompt change, every model swap, every retriever change, and every chunking change — at minimum. Mature teams wire RAGAS or a similar framework into CI so that a pull request modifying any of those surfaces automatically reports component-level deltas against the golden set. A weekly or per-release full sweep on a larger held-out set catches drift that the small CI set might miss.

What's the difference between faithfulness and correctness?

Faithfulness measures whether the answer is supported by the retrieved context — it ignores whether the context itself is correct. Correctness measures whether the answer matches ground truth. A model can be perfectly faithful to a wrong document (high faithfulness, low correctness) or invent a correct fact not present in the context (low faithfulness, high correctness). Production systems usually optimize faithfulness first because hallucinations are the highest-trust-cost failure mode, even when the hallucinated answer happens to be right.

Should I use an LLM-as-judge or human raters?

Both, in different volumes. LLM judges are cheap enough to run on every CI build and every production sample. Human raters catch systematic judge errors — for example, judges that over-rate verbose answers or under-rate concise ones. A typical setup is LLM-judge for all eval runs plus a quarterly calibration audit where humans re-score a few hundred examples and you measure agreement. If judge-human agreement drops below 0.7 Cohen's kappa, retune the rubric.

How do I know when my RAG system is "good enough" to ship?

No universal threshold, but a useful rule of thumb is faithfulness above 0.9, answer relevancy above 0.85, context recall above 0.8 on a representative golden set, plus p95 latency and cost-per-query within product constraints. Beyond the numbers, run a dogfood week with the target user persona — internal users catch UX failures (refusals, hedging, formatting) that no metric captures. Ship when both the numbers and the dogfood feedback are green.