A retrieval assistant that answers wrongly is almost never suffering from a model problem. In the knowledge assistants we build, the cause sits in the content, the chunking or the index in the large majority of cases, and the diagnostic order that finds it fastest is to look at what was retrieved before looking at what was answered.
That order matters because the two failure modes look identical from the outside. An assistant that fetched the wrong passage and an assistant that fetched the right passage and ignored it both produce a confident, plausible, incorrect answer. Without retrieval logging you cannot tell them apart, and teams end up changing prompts to fix content problems.
What follows is six recognisable symptoms, the cause behind each, and what to do about it.
Why is the content the usual culprit?
Because a retrieval system inherits every flaw in the material it points at, and most organisations have never audited that material for this purpose.
Documentation is written for people who can skim, ask a colleague and notice that a page is dated. A retrieval system does none of those things. It treats a superseded procedure and the current one as equally authoritative, because nothing in the text says which is which. It reads a table without its heading as a set of unlabelled numbers. It cannot tell that a product page describes a discontinued line.
RAND’s study on the root causes of failure for AI projects, based on interviews with sixty-five experienced practitioners, names inadequate data among the leading causes of failure across the board. In retrieval work that finding is unusually literal, because here the data is simply the documents you already have.
This is why the content work is the real work. On a customer-facing assistant we delivered, the build began with a structured analysis of website copy, FAQ entries, product pages, technical sheets, brochures and historical customer service exchanges, before anything was indexed. That analysis is not preparation for the project, it is a large part of the project, and skipping it is the most common reason a pilot that impressed everyone in week two disappoints in week eight.
Which symptom points at which cause?
A confident but wrong answer usually means retrieval came back with nothing relevant and the model filled the gap. This is the symptom people call hallucination, and treating it as a model defect leads nowhere. The fix is a grounding check: if the retrieved context does not support an answer, the system should say so and hand off rather than produce fluent text. Designing that path is one of the highest-value decisions in the whole build.
The right document but the wrong passage points at chunking. Either the chunk boundary split an idea from the context that made it meaningful, or the chunk is so large that the relevant sentence is diluted among four unrelated topics. Tables separated from their headings are the classic case, because the numbers survive and the meaning does not.
An outdated answer means a superseded source is still in the index. This is an index lifecycle problem rather than a retrieval problem, and it needs a refresh schedule and a removal process, not better search. Organisations consistently plan the ingestion and forget the deletion.
Two answers that contradict each other means the corpus contains both versions and retrieval is picking whichever matches the phrasing of the question. No amount of prompting resolves this, because the system is faithfully reporting what it was given. The fix is editorial: decide which document is authoritative and remove or mark the other.
Works in testing, fails in production almost always means the test questions were written by the person who built the system. Builders write questions they know the answer to, phrased the way the documentation phrases it. Real users ask about the thing next to the thing, in their own words, with a typo. Historical support tickets are the best available source of realistic test questions precisely because nobody invented them.
Failure on combined questions means a single retrieval pass cannot join two facts that live in different documents. Asking what the warranty is on the product a customer bought last year requires two lookups and a join, and a straightforward retrieval setup does one lookup. This is a design limitation rather than a bug, and it is worth deciding early whether your questions need it, because the architecture differs.
In what order should you check?
Retrieval first, content second, prompting third, model last. That order is the opposite of the instinct, which is to change the prompt because it is the cheapest thing to change.
Start by logging the retrieved passages alongside every answer. Without that log, diagnosis is guesswork, and adding it later means discarding the failure examples you already collected. With it, most cases resolve in a minute: either the correct passage is in the log or it is not, and that single fact separates a content or retrieval problem from a prompting problem.
If the passage was never retrieved, look at the content before you look at the search configuration. Ask whether the answer exists in the corpus at all, whether it exists in more than one version, and whether it is phrased in vocabulary a user would never use. Many apparent retrieval failures are cases where the answer simply is not written down anywhere, which no amount of tuning will fix.
Gartner names unclear business value among the top reasons agentic AI projects get cancelled, and an assistant whose failures are never diagnosed becomes exactly that: something nobody can defend because nobody can explain it.
Only when the right passage is being retrieved and the answer is still wrong does prompting become the suspect. And only when a correct, well-prompted context still produces a bad answer is the model itself worth questioning, which in practice is rare.
How do you measure whether it is actually getting better?
Separate retrieval quality from answer quality, because a single accuracy number hides which half is broken.
For retrieval, the question is whether the passage containing the answer appears in the results at all. That can be measured against a set of questions with known source passages, and it improves through content work and chunking rather than through prompting. For answer quality, the question is whether the response is correct and complete given what was retrieved, which is a different measurement and improves through different changes.
Assemble the evaluation set from real questions, ideally fifty to two hundred, with the awkward ones deliberately included. That set is the most valuable artefact in the project, because it turns every subsequent change from an opinion into a comparison. Building it is work for whoever owns the content rather than for the engineer, since only they know which questions are genuinely hard.
One measurement that gets skipped and should not: how often the assistant declines to answer. A rising refusal rate is not necessarily a regression, it can mean the grounding check is working. A refusal rate of zero, on a corpus that does not cover everything, means the system is answering questions it should not.
How much of this is architecture rather than tuning?
More than most teams expect, and it is worth separating three layers when you design.
The knowledge layer is the content, the chunking and the index. The conversation layer is how the assistant interprets intent and how it sounds. The hand-off layer is what happens when it cannot help, whether that is a route to a person, a form or a phone number. Keeping these separate makes diagnosis dramatically easier, because a tone complaint and an accuracy complaint stop being the same ticket.
It also prevents a specific and common failure: an assistant that gives technically correct answers that nobody acts on. If a customer receives the right information and still does not know what to do next, the knowledge layer worked and the hand-off layer was never designed. That is a product decision, not a retrieval one.
The single change worth making first
Log the retrieved passages next to every answer, starting today. It costs almost nothing, it changes diagnosis from speculation into a two-minute check, and it is the one thing that cannot be added retroactively to the failures you have already seen.
Everything else in this article is easier once that log exists, because every symptom above becomes a question you can answer by looking rather than a theory you have to test.
From wrong answers to a reliable RAG assistant
A reliable RAG assistant starts with the right knowledge, not necessarily a better model. By improving content quality, retrieval, and evaluation, organisations can reduce incorrect answers and build AI assistants that users can actually trust.
At DataNorth AI, we help organisations achieve this through our RAG Readiness Assessment and custom RAG Development services. From identifying weaknesses in existing knowledge bases to developing, integrating, and monitoring AI assistants, we focus on solutions that deliver accurate, relevant, and reliable answers.
Struggling with incorrect answers from your AI assistant? Get in touch with DataNorth AI to explore how we can improve your existing solution or build one that works reliably from the start.
Frequently asked questions about RAG accuracy
Why does a RAG assistant hallucinate?
Usually because retrieval returned nothing relevant and the model answered anyway. A language model asked a question with no supporting context will still produce fluent text. The fix is a grounding check that refuses to answer when retrieval comes back empty, not a better model.
How do you diagnose a wrong RAG answer?
Look at what was retrieved before you look at what was answered. Log the retrieved passages for every response. If the right passage was never fetched, the problem is retrieval or content. If it was fetched and ignored, the problem is prompting or context length.
What chunk size works best for RAG?
There is no universal size, but the useful rule is that a chunk should contain one complete idea plus enough surrounding context to be understood alone. Chunks that split a table from its heading, or bundle five unrelated topics, cause more wrong answers than any model choice.
How do you test a RAG assistant properly?
Use real questions from real users, not questions written by the person who built it. Historical support tickets and customer service transcripts are the best source, because they contain the phrasing and the awkward cases that a builder would never invent.
Should a RAG assistant ever say it does not know?
Yes, and designing that path is one of the highest-value decisions in the build. An assistant that admits uncertainty and hands off to a person keeps trust. One that always produces an answer trains users to verify everything, which removes the point of having it.
