RAG Evaluation Checklist

Build an evaluation file with questions, expected source documents, acceptable answers, and examples where no answer should be returned. Include recent documents, old documents, duplicate documents, and documents with conflicting claims.

Measure recall at k before judging answer quality. If the right passage never reaches the model, prompt engineering will only hide the real problem.

Chunking decisions are product decisions. A support bot may need short policy sections; a legal review tool may need larger context blocks with section headers and document metadata.

Review failed queries by looking at retrieved chunks. Common failures include missing titles, broken tables, boilerplate crowding out the useful passage, and embeddings that treat two product versions as interchangeable.

A RAG answer without citations is hard to audit. Ask the system to cite exact documents, show confidence, and admit when the retrieved context does not support an answer.

Use a separate check for groundedness: every material claim should map back to retrieved evidence. This matters more for customer support, healthcare, finance, and enterprise knowledge bases.

Documents change, product names change, and teams forget to remove stale content. Track index freshness, failed retrievals, low-confidence answers, and user corrections.

A mature RAG system has a data owner, an indexing schedule, access-control checks, and a way to remove revoked or sensitive content quickly.