Why most RAG prototypes fail in production

A retrieval-augmented generation prototype is easy to love. You point it at a folder of documents, ask the questions you already know the answers to, and it answers them well. The demo is honest about what it is: a system that works on the questions it was built for.

Production is a different system wearing the same name. It takes questions nobody rehearsed, from people who phrase things their own way, and every answer has a cost, a latency and a way to be wrong. Most prototypes fail here not because the model is weak, but because everything around the model was never built.

The demo gap

A demo optimises for one thing: a convincing answer to a known question. Production has to hold several things at once: the right passage retrieved, an answer grounded in it, a refusal when the documents do not contain the answer, a cost per answer you can afford, and a latency a person will wait through. A prototype that nails the first and ignores the rest looks finished and is not.

The failure is rarely dramatic. The system keeps answering; it just answers a quietly growing share of questions wrong, and nobody notices until a user does.

Retrieval is the hard part

The language model gets the attention, but in RAG the model is mostly downstream of a retrieval problem. If the right passage is not in the context, no amount of prompting recovers it; the model will answer confidently from whatever it was given.

The parts that break in production are the unglamorous ones:

• Chunking. How a document is split decides what can ever be retrieved together. Split too small and an answer is spread across chunks that never co-occur; too large and the signal drowns.
• Keyword versus meaning. Pure vector search misses exact terms: a part number, a clause reference, a name. Pure keyword search misses paraphrase. Production usually needs both, hybrid, then reranked.
• The long tail. The questions that matter most are often the rare ones, and those are exactly the ones a prototype tested on common questions never saw.

Evaluation is the thing you skipped

The reason a prototype cannot tell it is failing is that it has no way to measure itself. There is no eval set, so “is it good?” is answered by whoever last tried it.

Production needs an eval set: real questions, with what a correct answer looks like, run on every change. It does not have to be large to be useful: a few hundred questions drawn from real contracts or tickets will catch most regressions. What matters is that a change to a prompt, a model or a chunking rule produces a number that moves, not a vibe that shifts.

Once the eval set exists, the questions change shape. Not “does it work?” but “what fraction of answers are grounded in a cited source, and is that fraction going up or down?”

Cost and latency are features, not afterthoughts

A demo runs a handful of times, so nobody counts the tokens or the seconds. At a thousand questions a day, both are line items. A reranking step that adds a second is a second every user waits; a larger context that lifts accuracy a point may double the cost per answer.

These are product decisions, and they belong in the open: a target cost per answer, a latency budget, and a conscious trade when they conflict. A system without those targets does not fail loudly; it just quietly costs more and runs slower than anyone decided it should.

Refusal is a feature

The most expensive failure in production is a confident wrong answer, because it is the one a user acts on. A prototype almost never refuses; it was built to answer. A production system has to know when the retrieved passages do not support an answer and say so, and that behaviour has to be measured like any other: how often it refuses when it should, and how often it refuses when it should not.

What production-ready actually means

A RAG system is ready for production when you can say, with numbers:

• what fraction of answers are correct and grounded in a cited source;
• how often it refuses, and whether those refusals are the right ones;
• the cost per answer and the latency at the 95th percentile;
• and that all of the above are checked automatically whenever the system changes.

None of this is exotic. It is ordinary engineering, measurement and budgets and tests, applied to a part of the stack that is easy to treat as magic. The prototype proved the idea can work. Production is the work of proving it keeps working, on questions you have not seen yet, at a cost you chose on purpose.