Evaluating hallucination Mitigation in Retrieval-Augmented Generation Systems through Guarded Response Strategies, Embedding Comparison, and Retrieval Depth Analysis

Retrieval-Augmented Generation systems are widely used to improve the factual reliability of large language model outputs by grounding responses in retrieved external documents. However, despite retrieval grounding, RAG systems can still generate unsupported or partially supported claims, especially when retrieved context is incomplete, irrelevant, or semantically weak. This paper presents a structured evaluation of hallucination behavior in RAG systems under different embedding models, retrieval depths, and response-generation strategies.

We compare baseline generation, where the model answers regardless of retrieval confidence, with a guarded response strategy that selectively refuses to answer when the retrieved evidence is insufficient. The study evaluates three embedding models: BGE, E5, and MiniLM, across retrieval depths of k = 1, 3, and 5. A JSON-based structured hallucination scoring framework is used to measure supported answers, hallucination rate, refusal rate, grounded precision, and average judge confidence.

Results show that guarded generation reduces unsupported claims and improves reliability, but introduces a measurable precision-refusal tradeoff. The results indicate that hallucination prevention in RAG is not just about the retrieval process but also involves response control and evaluation as well. This study provides practical insights for deploying reliable RAG systems in critical settings where factuality, refusal behavior, and answer coverage must be balanced.