S
Sunilkumar Reddy Eraganeni
Guest
The demo that lied to everyone.
Many RAG projects begin the same way. Someone indexes a few dozen clean documents, asks five questions they already know the answers to, and the room goes quiet because the thing works. A budget gets approved. Somebody says "we should roll this out to the whole company by Q3."
Then it meets the real corpus. Think of a mid-sized bank with years of policy PDFs, three generations of intranet pages, wiki articles nobody has touched since a reorganisation, and spreadsheets pasted into slide decks. Pilot feedback is brutal. The assistant quotes a leave policy that was replaced last spring. It cannot find a form when you type its exact code. It answers a question confidently from a document the user was never supposed to see.
The instinct is to blame the model, the prompt, or both. I think that instinct is wrong more often than not, so the model deserves to be the last suspect, not the first.
What the pipeline really looks like
Most teams build some version of the following.
Documents flow from source systems into an ingestion step, get split into chunks, turned into embeddings, and stored in a vector index. At query time, the question is embedded, the closest chunks are fetched, stuffed into a prompt, and a language model writes the answer.
Notice how much of that diagram has nothing to do with the model. The whole top row is data engineering. That matters, because the people building RAG systems are often application developers who have never had to babysit a pipeline that ingests messy documents every night. Data engineers have, and most of our scars apply here.
Five layers where things go wrong.
Here is the map I use when someone tells me their RAG system "hallucinates." Usually it does not. It was handed the wrong evidence and did its best.
The data layer. Duplicates, outdated versions, scanned PDFs that contain no extractable text, and documents with no owner. If two versions of a policy sit side by side in the index, the retriever has no way of knowing which one is current. It simply returns whichever one sounds closer to the question.
The ingestion layer. Parsers are quietly terrible at tables. A pricing table flattened into one run-on line is useless, and nothing throws an error when it happens. Headers get detached from their sections, so a chunk that says "the limit is 5,000" no longer says what limit, for whom, or under which product.
The retrieval layer. Pure vector search is wonderful at meaning and surprisingly poor at exact strings. Search for "Form 7B" or a ticket ID, and you may get passages that are vaguely about forms instead. Many teams add keyword search alongside vectors for exactly this reason. Access control also lives here, and I will return to it below.
The generation layer. Even with good context, a model may ignore it, blend it with its own memory, or cite the wrong passage. This layer is real. It is just rarely where the first fire starts.
The operations layer. No test set, no alerting when quality drifts, costs creeping up, and no way to roll back a bad index. This is the layer that turns a one-off bug into a standing problem.
Why a single accuracy number misleads you.
Here is a trap that catches a lot of teams. A team builds a quick evaluation, asks the model to grade its own answers, gets "82% good," and ships. Six weeks later, users complain, the number says 81%, and nobody can say what changed.
An end-to-end score tells you that something is wrong. It cannot tell you where. If the answer was bad, was the right document never indexed, never retrieved, retrieved but ranked too low, or retrieved and ignored? Each of those has a different owner and a different fix.
Grading by a language model is useful, but it is biased in ways that are hard to measure, so I treat it as a screening tool and not a verdict. A modest batch graded by a human who knows the domain often teaches more than thousands graded by a machine. Ideally you do both and watch where they disagree.
A four-layer evaluation framework.
This is the setup I would put in place on day one of a serious RAG project. It is deliberately boring, because it borrows from how data teams already test pipelines.
Start with the golden set.
Everything depends on this. A golden set is a collection of real questions, each paired with the passage or passages that should answer it, plus a reference answer approved by someone who knows the subject. I would aim for somewhere between 150 and 300 questions to begin with. That is a judgement call and not a magic number. Fewer than about 100 and the scores wobble too much to trust. Many more and nobody maintains them.
Where do the questions come from? Not from the engineers. Pull them from help desk tickets, search logs, and a few sessions sitting next to the people who actually ask. Keep the ugly ones: typos, old product names, and questions the documents cannot answer at all. That last group matters, because a good system should say "I could not find that" instead of improvising.
Store the golden set in version control, like code. When a source document changes, update the expected answer in the same pull request.
Layer 1: Data Quality
Before anything touches an embedding, measure the input. Track freshness lag (how long between a source changing and the index reflecting it), parse success rate, duplicate ratio, and how many chunks carry complete metadata such as owner, date, and document type. Standard data quality checks, applied to documents instead of tables.
Layer 2: Retrieval
Given a golden question, did the right passage come back? Recall at k answers the plain version of that. Ranking metrics such as MRR or nDCG tell you whether it came back near the top, which matters because most prompts only have room for a handful of chunks. Add two checks people forget, namely whether filters (region, department, date) were applied correctly, and whether exact identifiers are found reliably.
Layer 3: Generation
Now, and only now, look at the model. Faithfulness asks whether every claim in the answer is supported by the retrieved text. Relevance asks whether the answer addresses what was asked. Citation accuracy asks whether the quoted source actually says what the answer claims. And refusal behaviour asks whether the system declines when the evidence is not there.
Layer 4: System behavior
Finally, the dull but decisive numbers. Latency at the 95th percentile, cost per answer, thumbs-down rate, and how often users escalate to a human. A slow system gets abandoned as surely as an inaccurate one.
A small sketch makes the retrieval check concrete.
Code:
def recall_at_k(golden, retriever, k=5):
hits = 0
for item in golden:
results = retriever.search(item["question"], top_k=k)
returned_ids = {r.doc_id for r in results}
if returned_ids & set(item["expected_doc_ids"]):
hits += 1
return hits / len(golden)Run it on every change to chunking, embeddings, or ranking. If the number drops, the problem lives in retrieval, not the prompt, and that knowledge alone saves days.
A production architecture worth building.
Once evaluation is in place, the architecture practically designs itself. The goal is the same pipeline from Figure 1, with the controls that data engineers usually add without being asked.
Change data capture over nightly dumps. If the index only refreshes once a day, you will answer today's questions with yesterday's policy. Event-driven updates cost more to build but close the freshness gap. For an archive of historical reports, a nightly batch is fine.
A quality gate before indexing. Reject or quarantine documents that fail parsing, contain obvious duplicates, or carry personal data that should never be searchable. Cleaning at the door beats explaining a leak later.
Access control at retrieval time. This is the one that scares security teams, and fairly. If the index holds the HR folder and the finance folder together, the retriever must filter by who is asking before the model ever sees a passage. Filtering the final answer is too late, because the model has already read the text. Store permission metadata with each chunk, and test the filter with the golden set. A question from a user without access should return nothing, and you should have a test that proves it.
Hybrid retrieval plus a reranker. Combine vector and keyword search, then let a smaller model reorder the shortlist. It costs a little latency and removes many near misses.
Treat the index like a production dataset.
One habit changes more than any model upgrade. Never re-embed documents in place. When you change the embedding model, the chunking rules, or the parser, build a second index beside the live one, run your golden set against both, and only switch traffic if the new one is at least as good.
Most vector databases support some form of alias or collection pointer, so the swap is a one-line change and rollback takes seconds. The alternative is a half-finished re-embedding that leaves the live index in a mixed state, with nobody able to say which chunks came from which model. That kind of silent corruption can eat weeks, and it is avoidable.
What this framework will not fix.
Evaluation sets age. Products get renamed, and a golden set from January will quietly stop representing October. Review it quarterly and keep adding questions from real failures.
Some questions are simply hard. Anything requiring reasoning across several documents, or arithmetic over a table, will stress even a well-built system. Retrieval tweaks only go so far, and for some use cases a structured query against a real database beats any amount of RAG.
And there is a cost. Shadow indexes, golden sets, access metadata, and monitoring all take engineering time. A small internal tool with ten users and low stakes probably does not need the full treatment. The framework scales down. Start with the golden set and the retrieval metric, since those two give most of the value for the least effort.
The Real Lesson
When a RAG system fails in production, the model rarely deserves the blame. The failure usually comes from the same causes that sink any data product, namely unclear ownership, unmeasured quality, and changes shipped without a way back. Those are problems data engineers already know how to solve.
So here is the reframing. Stop thinking of RAG as a clever prompt on top of a database. Think of it as a data pipeline whose final consumer happens to be a language model, and hold it to the standards you would hold any pipeline that feeds decisions. Teams that do this tend to ship systems people keep using. The rest keep tuning prompts and wondering why nothing improves.