Applications of Foundation Models
RAG Evaluation Metrics
RAG evaluation metrics separate retrieval quality from answer quality. Context recall checks whether retrieval found all needed evidence, context precision checks whether retrieved chunks avoid irrelevant noise, faithfulness checks whether the answer is grounded in the retrieved context, answer relevance checks whether the response addresses the question, and MRR checks whether the first relevant result ranks near the top.
Concepts
- Context recall measures whether retrieval captured all relevant chunks needed for a complete answer; low recall means key details are missing.
- Context precision measures whether retrieved chunks are actually relevant instead of irrelevant noise; high precision means clean, focused retrieval.
- Faithfulness measures whether the generated answer is supported by the retrieved context; low faithfulness means the model added unsupported claims.
- Answer relevance measures whether the final response directly addresses the user's question.
- Recall is about completeness, precision is about relevance versus noise, and faithfulness is about grounding.
- Mean Reciprocal Rank (MRR) measures how high the first relevant result appears in ranked retrieval results using 1 divided by the rank of the first correct result; it evaluates retriever ranking quality, not completeness or faithfulness.
- Recall and precision appear in both classification and RAG: classification recall and precision judge predicted labels, while context recall and context precision judge retrieved chunks, but both express the same idea of completeness versus signal-to-noise.
Exam tips
- Use context recall for the question did we find everything needed for a complete answer.
- Use context precision for the question did retrieval avoid irrelevant noise.
- Use faithfulness when the answer makes claims that are not grounded in retrieved context.
- Use MRR when the concern is whether the first relevant or best result appears near the top, not whether every relevant chunk was retrieved.
- If a question contrasts a classification metric with a RAG retrieval metric, map completeness to recall and relevance-versus-noise to precision in both domains.