Observability for features that are sometimes wrong

A retrieval or generation feature can be completely broken and perfectly green. The request succeeded, the latency was normal, no exception was thrown, and the answer was wrong.

Conventional observability was built for systems that fail loudly. This class fails quietly, and needs signals of its own.

What to log per call

More than you would for an ordinary endpoint, because reproduction is otherwise impossible:

• The full prompt as sent, including the retrieved context. Not a template reference — the actual text, or you cannot reproduce anything.
• A prompt version identifier. Every change gets one. Without it you cannot attribute a regression to a deployment.
• Model and parameters — name, version, temperature. Provider-side model updates are a real source of behaviour change you did not cause.
• Token counts and cost, in and out.
• Latency, split into retrieval, generation and post-processing. They move independently and for different reasons.
• The outcome, which is the part everyone omits.

That log contains user content, so it needs the same retention, access control and deletion path as any other store of user data. Deciding that after you have six months of it is much harder than deciding it now.

The four proxies for quality

You rarely have ground truth in production. You do have behaviour, and it is a good signal:

1. Acceptance. Did the user keep the output, or did they discard it? For a suggestion, whether they applied it. This is the closest thing to a direct quality measure you get for free.

2. Edit distance. When the user modifies the output before using it, how much? A rising edit distance is a quality regression that no error rate will show.

3. Retry and regeneration. A user asking again, rephrasing, or pressing regenerate is telling you the first answer failed. Regeneration rate is one of the most sensitive signals available.

4. Abandonment. They asked and then left without using anything. Ambiguous individually, meaningful in aggregate and over time.

None of these is precise. All of them move before your users start writing in.

The rates that catch specific failures

• Refusal rate. A model declining to answer. A jump usually means a prompt change or a provider-side update, and it is invisible to error monitoring because a refusal is a successful response.
• Fallback rate. How often you served the non-model path. A rise means the feature is degrading in a way users may not notice and you should.
• Schema violation rate, where output is structured. This is the closest thing to a conventional error in this stack.
• Empty retrieval rate. Queries where the retriever returned nothing relevant. Often the actual cause of a "the model is worse today" report.
• Truncation rate. Responses cut off at the token limit, which reads to a user as the model being confused rather than the limit being reached.

Alerting on distributions, not thresholds

A single wrong answer is not an incident; a shifted distribution is. Alert on:

• Week-over-week change in acceptance, regeneration and refusal rates.
• Percentile latency rather than the mean, because generation latency has a long tail that the mean hides entirely.
• Cost per request by prompt version, which catches a context change that silently tripled the input.

And treat a provider model update as a deployment: pin versions where you can, and run the evaluation set when a pin moves.

A feature that returns a confident, well-formed, wrong answer is a 200. If your dashboard cannot distinguish that from a correct one, it is not measuring the feature.

Close the loop back to evaluation

Production logs are the best source of evaluation cases you will ever have, because they are real:

1. Sample the calls with low acceptance or high edit distance.
2. Have someone label a manageable number of them each week.
3. Add the clear failures to the evaluation set.
4. Re-run the set on every prompt change.

That loop is what turns "it feels worse this week" into a number, and it is the only durable way to keep a feature from drifting.