Applications of Foundation Models
RAG Evaluation Metrics
RAG metrics separate retrieval quality from answer quality. Context recall measures completeness, context precision measures retrieval noise, faithfulness measures grounding, answer relevance measures whether the response addresses the question, and MRR measures how highly the first relevant result ranks.
Concepts
- Context recall measures whether retrieval captured all relevant chunks needed for a complete answer; low recall means key details are missing.
- Context precision measures whether retrieved chunks are actually relevant instead of irrelevant noise; high precision means clean, focused retrieval.
- Faithfulness measures whether the generated answer is supported by the retrieved context; low faithfulness means the model added unsupported claims.
- Answer relevance measures whether the final response directly addresses the user's question.
- Recall is about completeness, precision is about relevance versus noise, and faithfulness is about grounding.
- Mean Reciprocal Rank (MRR) measures how high the first relevant result appears in ranked retrieval results using 1 divided by the rank of the first correct result; it evaluates retriever ranking quality, not completeness or faithfulness.
- Recall and precision appear in both classification and RAG: classification recall and precision judge predicted labels, while context recall and context precision judge retrieved chunks, but both express the same idea of completeness versus signal-to-noise.
Exam tips
- Use context recall to ask whether retrieval found everything needed for a complete answer.
- Use context precision to ask whether retrieval avoided irrelevant noise.
- Use faithfulness when the answer makes claims that are not grounded in retrieved context.
- Use MRR when the concern is whether the first relevant or best result appears near the top, not whether every relevant chunk was retrieved.
- If a question contrasts a classification metric with a RAG retrieval metric, map completeness to recall and relevance-versus-noise to precision in both domains.