TRIDOMRAG-BENCH: A Tri-Domain Benchmark for Evaluating Retrieval-Augmented Generation in Healthcare, Legal, and Finance

Abstract Retrieval-augmented generation (RAG) now powers most knowledge-intensive NLP pipelines, yet the frameworks used to evaluate these systems have not kept pace with the technology. A diversity of domains is treated as one-size-fits-all in current benchmarks, but it is not. For instance, the RAGAS metrics are generic and use natural language inference (NLI) scoring completely.

It does not takes care of the numerical accuracy, which is a key consideration in financial establishments. In addition, there is no existing benchmark that assesses the same RAG system in healthcare, legal, and financial sectors through a single unified lens. In this paper, we introduce TriDomRAG, the first benchmark to assess RAG systems in these three important areas under a common protocol.

We propose a composite score, the Domain Hallucination Score (DHS) which combines faithfulness, attribu- tion, and terminology which assess these three characteristics and technical terminology via domain calibrated weights. We used 1200 question-answer pairs for our benchmark. We draw our datasets from experts, such as PubMedQA (healthcare), Open Australian Legal QA (legal), and a merged FinanceBench and 10-K corpus (finance).

With the use of GPT- 4o-mini, we tested six different RAG configurations and observed that dense retrieval with chunk-level retrieval and rewriting yields the best results. Dense chunk retrieval achieves a DHS of 0.585, outperforming BM25 by 1.6 percentage points. Our results emphasize that real data is not something that can be compromised; e.g., substituting synthetic legal data with legal corpora from Australia improved scores for attribution from almost zero to 0.678.

Lastly, we discuss the well-known shortcomings of conventional RAGAS metrics in the financial domain and present a numerical proximity extension of the DHS. All code and data have been released as open source at https://github.com/abhi6579/tridomrag-bench.