Client project, anonymised

The AI feature worked in the demo and broke in production

The feature passed its demo, passed review, and shipped. Then real users asked real questions, the answers started to drift, and nobody could tell whether any fix made it better, because nothing was being measured.

A project our founder led before starting Keelwork. The client is not named, under NDA; the story, the architecture and the numbers are real.

Service: Evals, cost and latency
Stack: Eval set from production, Tracing + observability, CI gates, Cost + latency budgets

Most AI features die the same way. They work in the demo, pass review, and ship. Then real users ask questions nobody rehearsed, the answers drift, the bill climbs, and the team cannot tell whether any fix helps, because nothing is measured. This is the sprint that turns that feature from a liability back into something you can change on purpose.

What changed over one sprint, on a feature already in production:

92%answer accuracy, now measuredwas not measured at all
2.3sp95 latency, from 4.1sback inside the 2.5s budget
35%lower cost per answerafter prompt and model changes
240eval cases gating every changewas zero

It passed review, then it met real users

The feature did everything right on the way out. It answered the test questions well, a reviewer tried it, and it shipped. The demo was honest about what it was: a system that worked on the questions it was built for.

Production sent the rest. Questions phrased in ways nobody tried, about edge cases nobody listed, at a volume nobody load-tested. Some answers were quietly wrong. The p95 latency was twice what anyone remembered from the demo. The monthly bill kept climbing. And every attempt to fix one problem risked breaking another, because there was no way to tell.

The team did not have a model problem. They had a measurement problem.

You cannot fix what you cannot see

Two things were missing, and they are the two things that separate a demo from a system.

There was no eval set, so “is it good?” was answered by whoever last tried it, on whatever question they happened to pick. A prompt change that fixed one complaint could regress ten other cases silently, and the first anyone heard of it was a new complaint.

There was no observability, so when an answer was wrong, there was no way to see what the model was given, what it retrieved, how many tokens it burned, or how long it took. Debugging was re-running the question by hand and hoping it failed the same way.

So the first move was not to touch the model. It was to make the feature visible.

Instrument first, change nothing yet

Before any fix, every request gets traced: the input, the retrieved context, the full prompt, the model’s output, the tokens, the latency, and the cost. Those traces flow to one place, which feeds both the dashboards the team watches and the eval set we are about to build. Only once we could see the feature did we start changing it.

The first week’s work. Make every request visible, then feed both the dashboards and the eval set from the same traces.

The eval set comes from production, not imagination

A good eval set is not a list of questions someone invented. It is the real distribution of what users actually ask, including the failures. We sampled traces across a representative span, pulled in the specific questions users had complained about, and labelled each with what a correct, cited answer looks like. The failures cluster, and the clusters are the map of what to fix first.

The failures, bucketed. A third of the wrong answers were the same bug, a stale document still being cited, which is a retrieval and freshness fix, not a model one.

With the clusters in hand, the fixes were specific. The stale-document bug was a reindex-on-change problem, not a model one. The wrong-passage cases were a retrieval and reranking fix. The over-long answers were a prompt and cost fix. None of them needed a bigger model.

A gate on every change

Once the eval set existed, it became the gate. Every change, a prompt edit, a model swap, a retrieval tweak, runs the full eval set in CI before it can merge. If any metric regresses against the last shipped version, the change is blocked, not shipped and discovered later.

The habit that keeps it fixed. The eval set, not a reviewer’s memory, decides whether a change is allowed to reach users.

Cost and latency, back under a budget

The traces made the cost and latency visible for the first time, and both had a clear cause. The p95 latency was dominated by a long generation step on a model heavier than most questions needed. The cost followed the same step. Routing the easy questions to a smaller model and tightening the answer format brought both back under budget, and the eval set confirmed accuracy held while they dropped.

The after, drawn. Latency back under its budget and cost down a third, with accuracy confirmed to hold by the same eval set.

What the traces kept catching

Instrumentation is not a one-time cleanup. The same dashboards and eval set keep earning their place:

Silent regressions. A model-provider update subtly changes behaviour. The eval set catches it on the next run, before a user does.
The slow creep. Cost or latency drifting up a few percent a week, invisible per request, obvious on a dashboard.
New failure shapes. A kind of question the system has never seen starts arriving. It shows up as a cluster of low scores, and becomes the next batch of eval cases.
The confident regression. A change that improves the average but quietly breaks a specific, important case. Per-case eval results catch what an average hides.

What a sprint like this leaves behind

The team keeps the eval set, the traces, the dashboards, and the CI gate, in their own stack. The feature that used to be risky to touch is now safe to change, because every change is measured before it ships. That is the difference between a demo and a system, and it is the part the client owns from the first day.