Evidence Coverage Evaluator: A Comprehensive Framework for Assessing Grounding Quality in Retrieval-Augmented Generation Systems

Retrieval-Augmented Generation (RAG) systems have emerged as a powerful paradigm for enhancing large language models with external knowledge. However, evaluating whether generated answers are properly grounded in retrieved evidence remains a critical challenge. Unlike accuracy-based metrics, evidence coverage evaluates whether each factual claim is explicitly supported by retrieved context, independent of surfacelevel correctness.

This paper presents the Evidence Coverage Evaluator (ECE), a comprehensive framework for assessing evidence coverage in RAG-generated answers. ECE introduces two complementary evaluation modes: Mode A, a lightweight NLIbased approach using pre-trained natural language inference models (RoBERTa-MNLI) achieving 0.88-0.91 coverage scores in 2.1-2.8 seconds, and Mode B, an LLM-based judge mode using local Ollama models (Mistral, Llama3, Gemma3, DeepSeek-R1) achieving 0.86 coverage scores with 100% success rates in 23-31 seconds for top-performing models. Our framework extracts finegrained claims from answers, retrieves relevant evidence passages using BM25 or embedding-based methods, and scores claimevidence pairs to compute coverage metrics.

Experimental results demonstrate that ECE outperforms baseline methods (keyword matching, BERTScore, direct NLI) by 10-32% in coverage assessment. Additionally, ECE includes advanced features such as citation quality assessment, unsupported claim identification, and actionable feedback generation. The framework is open-source, extensible, and designed for integration into CI/CD pipelines for continuous quality assurance in production RAG systems.