A
Anil Reddy Putlacheru
Guest
A developer queries an internal AI assistant. The assistant looks up the documentation, finds a relevant troubleshooting document, and provides an answer.
You get an incorrect answer.
This can seem like a problem with retrieval. Maybe the embedding model didn't find the correct document, or the index needs rebuilding.
But when you look at the retrieved documents, the correct document is already there.
This means you're potentially dealing with a reliability issue. The RAG system has found the relevant information, but it isn't giving you a reliable answer. Improving the search alone won't necessarily fix it.
The issue can occur after retrieval, when the application decides which information to trust, what it means, and whether there's enough evidence to answer the question.
A RAG system has several stages.
The application takes in a question, finds relevant information from documents, and passes it to a language model to generate an answer based on the context provided.
Every part of this process can affect the outcome.
Say you have an internal troubleshooting assistant with documentation for three environments: Development, Staging, and Production.
A user asks what to do when their tests fail in Staging.
The search returns a troubleshooting document with the correct information for fixing issues in Staging. It also returns a runbook with similar instructions for Production.
Both documents are relevant to the general topic, but only one applies to the user's situation.
If the assistant combines them, the answer may contain technically correct information but give the wrong advice.
The issue here isn't necessarily with the search engine. It's with the application not maintaining enough context to know which information to use.
Vector searches are designed to return semantically relevant information.
However, they won't necessarily return the most up-to-date, authoritative, or environment-specific information.
Say you have three documents in your knowledge base. All describe how to restart a service.
One is an up-to-date runbook. Another is an old troubleshooting note, and the third is an incident report.
A similarity search may return all three because they discuss the same service and symptoms.
The assistant then needs to decide which document should be treated as the source of truth.
For a well-designed RAG system, you'll need metadata such as environment, system, version, publication date, and document type.
This metadata should be used during retrieval wherever possible to narrow the results to information that applies to the user's request.
If someone asks about Staging, we wouldn't want to recommend Production instructions just because the error messages are similar.
This can't always be solved by writing a more persuasive prompt. Some restrictions need to be enforced in the retrieval and application layers.
Chunking is another common reason for misleading answers.
Long documents are often broken into smaller chunks. This makes them easier to embed and search, but important conditions can become separated from the instructions they apply to.
Say you have a runbook containing this instruction:
Later in the document, it says this should only be done in a non-production environment and only after confirming that no jobs are running.
If the retrieved chunk doesn't include those conditions, the model may recommend an action that isn't appropriate.
The wording in the chunk may be correct, but the information is incomplete.
Teams should test whether important conditions are preserved throughout document processing and retrieval.
For procedural documents, keeping headings, warnings, prerequisites and related steps together may be more useful than creating equal-sized chunks.
Where possible, chunk boundaries should follow the structure of the original document.
Say someone asks whether their integration tests are failing because of a recent deployment.
The assistant finds a deployment note and a few troubleshooting documents. None of them include the actual test execution logs.
The model may produce a plausible explanation based on similar issues, but there isn't enough evidence to confirm the cause.
A reliable assistant should distinguish between what is known and what remains uncertain.
It could respond with:
This answer is less certain, but it's more useful.
It tells the user what information is missing and what to investigate next.
RAG systems should be tested on whether they recognise when there isn't enough evidence to provide a reliable answer.
Adding citations can make generated answers more transparent.
However, citations aren't useful if the referenced passages don't support the claims.
The assistant may cite a document discussing the correct service even though that document doesn't support its conclusion.
For example, a troubleshooting document may mention a timeout as one possible symptom.
The assistant might cite that document and confidently state that the timeout was caused by a recent deployment.
The citation looks legitimate, but it doesn't establish the cause.
Your evaluation should check the relationship between each claim and its cited passage.
In practice, you'll want to know whether the source supports the claim, whether it applies to the requested environment, and whether the assistant ignored contradictory evidence.
The presence of citations and the correctness of those citations should be measured separately.
If your RAG assistant provides incorrect answers, you'll want to identify where the problem occurs.
This can be difficult if you're only testing the final response.
A useful approach is to break the evaluation into three areas:
Say you have a test question with a known answer and a verified supporting source.
If the source never appears in the search results, retrieval needs investigation.
If the source appears but gets ignored during context selection, the issue is in the intermediate processing.
If the source reaches the model but the answer contradicts it, the problem is with answer generation or how the model uses the evidence.
Each of these issues requires a different solution.
Increasing the number of retrieved chunks may help when important documents are missing, but it can make other problems worse by introducing more irrelevant information.
Your RAG evaluation should include scenarios that test the limits of the system.
Add questions involving outdated information, conflicting sources, and environment-specific conditions.
Include questions where no reliable source is available to support an answer.
Also add scenarios that check access controls. Just because a document is relevant doesn't mean the person asking the question is authorised to access it.
These tests help you understand whether the assistant can operate reliably in an enterprise environment.
Sometimes the correct response won't be a detailed explanation.
It may be to highlight conflicting information, ask for clarification, or explain that there isn't enough evidence to answer.
Say you have 100 test questions relating to troubleshooting procedures.
For each question, an engineer identifies the documents expected to be retrieved and the acceptable answer.
You then record whether the relevant evidence was found, whether the response used the correct sources, and whether its claims were supported.
You might find that most questions retrieve the correct documents, but the generated answers are not consistently grounded in those documents.
That would suggest improving vector search alone won't fix the main problem.
Instead, you could investigate context selection, source prioritization, and how the assistant handles conflicting information.
The 100 questions are only an illustrative example. In a real evaluation, you should report the actual results, how the dataset was built, and the criteria used to judge the answers.
Before releasing a RAG assistant to a wider audience, I'd want answers to a few questions:
These are practical checks that help determine whether a proof of concept can become a system that engineers can investigate, maintain, and rely on.
RAG can improve an AI assistant by connecting it to external information.
However, that information won't automatically be accurate, up-to-date, or appropriate for every question.
Your system may retrieve the correct document but still provide an incorrect answer because of missing conditions, conflicting sources, or unsupported claims.
That's why you need good document quality, appropriate retrieval filters, reliable context handling, and checks that verify whether the generated answer is supported by the evidence.
The goal isn't simply to retrieve more information.
It's to use the right information for the right question and recognise when there isn't enough evidence to give a reliable answer.
You get an incorrect answer.
This can seem like a problem with retrieval. Maybe the embedding model didn't find the correct document, or the index needs rebuilding.
But when you look at the retrieved documents, the correct document is already there.
This means you're potentially dealing with a reliability issue. The RAG system has found the relevant information, but it isn't giving you a reliable answer. Improving the search alone won't necessarily fix it.
The issue can occur after retrieval, when the application decides which information to trust, what it means, and whether there's enough evidence to answer the question.
You have multiple stages in a RAG system.
A RAG system has several stages.
The application takes in a question, finds relevant information from documents, and passes it to a language model to generate an answer based on the context provided.
Every part of this process can affect the outcome.
Say you have an internal troubleshooting assistant with documentation for three environments: Development, Staging, and Production.
A user asks what to do when their tests fail in Staging.
The search returns a troubleshooting document with the correct information for fixing issues in Staging. It also returns a runbook with similar instructions for Production.
Both documents are relevant to the general topic, but only one applies to the user's situation.
If the assistant combines them, the answer may contain technically correct information but give the wrong advice.
The issue here isn't necessarily with the search engine. It's with the application not maintaining enough context to know which information to use.
Irrelevant documents can be returned.
Vector searches are designed to return semantically relevant information.
However, they won't necessarily return the most up-to-date, authoritative, or environment-specific information.
Say you have three documents in your knowledge base. All describe how to restart a service.
One is an up-to-date runbook. Another is an old troubleshooting note, and the third is an incident report.
A similarity search may return all three because they discuss the same service and symptoms.
The assistant then needs to decide which document should be treated as the source of truth.
For a well-designed RAG system, you'll need metadata such as environment, system, version, publication date, and document type.
This metadata should be used during retrieval wherever possible to narrow the results to information that applies to the user's request.
If someone asks about Staging, we wouldn't want to recommend Production instructions just because the error messages are similar.
This can't always be solved by writing a more persuasive prompt. Some restrictions need to be enforced in the retrieval and application layers.
Chunking can cause information loss.
Chunking is another common reason for misleading answers.
Long documents are often broken into smaller chunks. This makes them easier to embed and search, but important conditions can become separated from the instructions they apply to.
Say you have a runbook containing this instruction:
Later in the document, it says this should only be done in a non-production environment and only after confirming that no jobs are running.
If the retrieved chunk doesn't include those conditions, the model may recommend an action that isn't appropriate.
The wording in the chunk may be correct, but the information is incomplete.
Teams should test whether important conditions are preserved throughout document processing and retrieval.
For procedural documents, keeping headings, warnings, prerequisites and related steps together may be more useful than creating equal-sized chunks.
Where possible, chunk boundaries should follow the structure of the original document.
Overconfident Answers
Say someone asks whether their integration tests are failing because of a recent deployment.
The assistant finds a deployment note and a few troubleshooting documents. None of them include the actual test execution logs.
The model may produce a plausible explanation based on similar issues, but there isn't enough evidence to confirm the cause.
A reliable assistant should distinguish between what is known and what remains uncertain.
It could respond with:
This answer is less certain, but it's more useful.
It tells the user what information is missing and what to investigate next.
RAG systems should be tested on whether they recognise when there isn't enough evidence to provide a reliable answer.
Check the citations.
Adding citations can make generated answers more transparent.
However, citations aren't useful if the referenced passages don't support the claims.
The assistant may cite a document discussing the correct service even though that document doesn't support its conclusion.
For example, a troubleshooting document may mention a timeout as one possible symptom.
The assistant might cite that document and confidently state that the timeout was caused by a recent deployment.
The citation looks legitimate, but it doesn't establish the cause.
Your evaluation should check the relationship between each claim and its cited passage.
In practice, you'll want to know whether the source supports the claim, whether it applies to the requested environment, and whether the assistant ignored contradictory evidence.
The presence of citations and the correctness of those citations should be measured separately.
Know where your issue is.
If your RAG assistant provides incorrect answers, you'll want to identify where the problem occurs.
This can be difficult if you're only testing the final response.
A useful approach is to break the evaluation into three areas:
- Retrieval quality: Did the search return the correct documents needed to answer the question?
- Context suitability: Were the documents current, applicable, and appropriate for the user's request?
- Answer grounding: Was the final answer supported by the retrieved evidence?
Say you have a test question with a known answer and a verified supporting source.
If the source never appears in the search results, retrieval needs investigation.
If the source appears but gets ignored during context selection, the issue is in the intermediate processing.
If the source reaches the model but the answer contradicts it, the problem is with answer generation or how the model uses the evidence.
Each of these issues requires a different solution.
Increasing the number of retrieved chunks may help when important documents are missing, but it can make other problems worse by introducing more irrelevant information.
Create test scenarios that show the limits of your RAG.
Your RAG evaluation should include scenarios that test the limits of the system.
Add questions involving outdated information, conflicting sources, and environment-specific conditions.
Include questions where no reliable source is available to support an answer.
Also add scenarios that check access controls. Just because a document is relevant doesn't mean the person asking the question is authorised to access it.
These tests help you understand whether the assistant can operate reliably in an enterprise environment.
Sometimes the correct response won't be a detailed explanation.
It may be to highlight conflicting information, ask for clarification, or explain that there isn't enough evidence to answer.
Example
Say you have 100 test questions relating to troubleshooting procedures.
For each question, an engineer identifies the documents expected to be retrieved and the acceptable answer.
You then record whether the relevant evidence was found, whether the response used the correct sources, and whether its claims were supported.
You might find that most questions retrieve the correct documents, but the generated answers are not consistently grounded in those documents.
That would suggest improving vector search alone won't fix the main problem.
Instead, you could investigate context selection, source prioritization, and how the assistant handles conflicting information.
The 100 questions are only an illustrative example. In a real evaluation, you should report the actual results, how the dataset was built, and the criteria used to judge the answers.
What I'd want to know.
Before releasing a RAG assistant to a wider audience, I'd want answers to a few questions:
- Can the application filter documents by environment and user access?
- Does it identify outdated information?
- Are warnings and prerequisites preserved during retrieval?
- Can it explain when there isn't enough evidence?
- Do the citations actually support the claims?
- Can engineers inspect the retrieved passages and identify where an incorrect answer originated?
These are practical checks that help determine whether a proof of concept can become a system that engineers can investigate, maintain, and rely on.
Key Takeaway
RAG can improve an AI assistant by connecting it to external information.
However, that information won't automatically be accurate, up-to-date, or appropriate for every question.
Your system may retrieve the correct document but still provide an incorrect answer because of missing conditions, conflicting sources, or unsupported claims.
That's why you need good document quality, appropriate retrieval filters, reliable context handling, and checks that verify whether the generated answer is supported by the evidence.
The goal isn't simply to retrieve more information.
It's to use the right information for the right question and recognise when there isn't enough evidence to give a reliable answer.