I kept running into the same problem while building RAG systems: everyone has an opinion on whether semantic chunking beats fixed size, or whether hybrid retrieval is worth the extra complexity, but almost nobody has actually measured it on their own corpus. So I built Retrieval Arena to answer that for myself instead of going off intuition. What it does: runs different chunking strategies, retrievers, and rerankers against the same golden eval set, scores retrieval quality and generation quality separately (precision, recall, MRR, nDCG for retrieval, correctness and faithfulness via an LLM judge for generation), and tracks latency and token cost per configuration. So instead of “I think hybrid is better,” I get an actual side by side comparison. A few things it’s already surfaced that I didn’t expect: pure vector search was consistently the weakest retriever on my corpus, BM25 held its own better than I assumed. Better retrieval also didn’t always mean better generated answers, I caught a case where the right chunk was retrieved fine but got dropped by my context budget before the generator ever saw it, which is invisible if you only track one end to end score. I’m posting mostly because I want pushback on the methodology, not just the code. Things I’m genuinely unsure about: is a 44 question golden dataset big enough to trust these comparisons, should I be running confidence intervals on top of the averages, is my LLM judge setup actually reliable or am I just trusting it too much. Repo’s here if you want to look at the actual eval design or the golden dataset construction: https://github.com/ayeangad/Retrieval-Arena Would genuinely appreciate anyone who’s built or evaluated a RAG system tearing into the approach, especially if you think I’m measuring the wrong things. submitted by /u/Whyrureadingthisz
Originally posted by u/Whyrureadingthisz on r/ArtificialInteligence