Why your AI answers are getting worse

Your AI feature can return successful API responses while producing worse answers. It can also degrade in ways ordinary monitoring catches, such as parsing errors or slower calls. Check both whether the service is running and whether its answers are useful.

When quality drops, investigate the whole request path. The model is one possible cause. Prompts, retrieval, tool responses, application settings, and the mix of users can also change.

TL;DR Compare saved requests and inputs separately from live data. Inspect model, prompt, source, and tool changes, then test the fix without changing customer records.

Record what produced each result

A useful trace identifies the model version, prompt version, relevant application release, retrieval configuration, source versions, and tool results. It should connect those details to the final task outcome.

Record enough to compare results without collecting unnecessary personal data or secrets. Restrict access and set retention rules. Logs that nobody can safely inspect are not a practical investigation tool.

If a provider offers a dated model snapshot, using it can reduce unplanned alias changes. It does not freeze the rest of the system or guarantee constant behavior.

Use an incident to understand the risk

Anthropic's September 2025 postmortem describes several infrastructure issues that degraded Claude responses. It is a concrete example of quality changing because of serving behavior rather than a planned new-model release.

It does not establish that every unexplained regression comes from the provider. Use it to justify observing results and retaining comparison evidence, not to skip your own investigation.

Separate fixed comparisons from live-data checks

Keep a fixed set of requests with saved versions of the source material and tool responses. Replaying it helps compare model or application changes without also changing the evidence the model sees.

Keep a second evaluation that uses current retrieval and tool data. If the fixed replay holds but the live-data evaluation worsens, investigate the source documents, integrations, permissions, and data freshness. If both sets of results get worse, inspect changes they share. These are diagnostic clues, not proof of one cause.

Then compare real traffic by request type and customer group. A lower overall success rate may reflect harder requests arriving, even when performance within each group is unchanged.

Score the property the product needs

Exact match works for some classification labels and structured fields. It is a poor sole measure for a useful prose answer that can be worded several ways.

Task	Useful quality check
Route a ticket	Correct destination and costly misroutes
Answer a policy question	Supported claims and applicable source
Extract a field	Correct value and valid structure
Complete a tool workflow	Verified external outcome
Decline an unsafe request	Correct refusal without excessive refusals

Use human review to check automated graders. OpenAI's evaluation best practices recommend evaluations matched to the task and continued evaluation as the application changes.

Choose alert thresholds from normal variation and the consequences of failure. There is no universal case count or detection window that makes every feature safe. A rare but serious permission failure deserves different handling from a small style regression.

Test a change without changing customer records

Run a candidate beside the current system using approved data, recorded tool responses, or isolated tools. Disable external writes in the comparison path. Shadow traffic must not send a second email or create a duplicate booking.

Compare quality and complete workflow cost, then review disagreements. Change one major variable at a time when diagnosing a regression, so the result remains interpretable.

OpenAI's deprecation notices are also part of release planning. A model retirement needs evaluation and migration time even when the replacement looks similar.

Make the alert actionable

An alert should identify the affected workflow, show representative failures, and name an owner. That person needs a way to disable risky actions, return to a tested configuration, or divert work to a staffed manual process.

Do not assume another provider is a drop-in fallback. It needs its own evaluation and capacity check. The same applies to a previous setup if the data or tools have changed since it last ran.

Start with a recent complaint and reconstruct the exact inputs behind it. If that is difficult, versioning and trace quality are the first fixes. Once you can reproduce the failure, add it to your answer-quality checks so future changes are tested against it.

Sources

• Anthropic: A postmortem of three recent issues
• OpenAI: Evaluation best practices
• OpenAI: Deprecations