The RAG Evaluation Framework Every Enterprise Needs

The Silent Degradation Problem

RAG systems don't fail dramatically. They degrade gradually. A document update introduces a formatting change that breaks your chunking logic. A model provider tweaks their embedding model and your retrieval accuracy drops 5%. A new category of user queries emerges that your system handles poorly. Each degradation is small enough to miss individually, but they compound.

The Four Evaluation Dimensions

1. Retrieval Quality

Is the system finding the right documents? Metrics:

• Precision@k — What percentage of the top-k retrieved chunks are actually relevant?
• Recall@k — What percentage of all relevant chunks appear in the top-k?
• Mean Reciprocal Rank — How high does the first relevant chunk appear?

Build a golden dataset of 300+ queries with annotated relevant passages. Re-run this evaluation weekly and after any system change.

2. Generation Faithfulness

Does the answer accurately reflect the retrieved context? This is the hallucination detection dimension. Use LLM-as-judge to evaluate whether each claim in the generated answer is supported by the retrieved passages. Track the faithfulness score over time — any downward trend indicates either retrieval degradation or generation quality issues.

3. Answer Completeness

Does the answer address all aspects of the question? A system might faithfully represent one retrieved passage while missing other relevant information. Completeness evaluation requires comparing the answer against all relevant passages, not just the ones that were retrieved.

4. End-to-End Correctness

Is the final answer actually correct? This is the ground truth evaluation that catches failures across all stages. It requires human-verified expected answers for a subset of your evaluation queries.

Measuring retrieval quality alone is like checking your car's fuel gauge but ignoring the engine temperature. You need visibility into every stage of the pipeline.

Continuous Evaluation Architecture

The evaluation pipeline runs in three modes:

1. CI/CD evaluation — Triggered on every code change, prompt update, or configuration change. Blocks deployment if quality drops below thresholds
2. Scheduled evaluation — Daily runs against the full golden dataset to detect gradual drift from external changes (model updates, document corpus changes)
3. Production sampling — Continuous sampling of live queries, evaluating a random subset in real-time to detect issues with real user behavior patterns

Building the Golden Dataset

The golden dataset is the foundation of your evaluation framework. Building it well matters more than any evaluation metric:

• Source queries from real user logs, not synthetic generation
• Include diverse query types: factual lookup, multi-hop reasoning, comparison, summarization
• Annotate relevant passages at the chunk level, not just the document level
• Include negative examples: queries that should return "not found" or "insufficient information"
• Version the golden dataset and update it quarterly as your document corpus evolves

Alerting and Response

Evaluation without alerting is reporting, not monitoring. Set thresholds for each metric and alert when they're breached. But more importantly, build runbooks for each alert type: retrieval degradation suggests chunking or embedding issues, faithfulness drops suggest prompt or model problems, and completeness drops suggest coverage gaps in your document corpus.