Why your AI feature is confidently wrong

A customer asks whether they qualify for a refund. The AI answers clearly, cites a policy, and gets the eligibility rule wrong. The problem is now a customer promise your team must resolve.

Before changing the model, identify how the wrong answer reached the user. Was the policy missing, outdated, retrieved incorrectly, or misread? Did the product require an answer when it should have asked a question? Each failure needs a different fix.

We would first narrow the feature's authority: what it may answer, which evidence it needs, and when it must ask for help. Then test whether a model change improves the remaining errors.

TL;DR Trace a wrong answer back to its evidence before changing the model. Check sources, enforce action limits in code, and give missing or conflicting information a clear route to clarification or review.

Define the evidence a correct answer needs

For account-specific questions, use the current system of record. For policy questions, use the policy that applies to that customer and date. A model's general knowledge cannot confirm the state of a particular order.

Record source ownership and update rules. If nobody maintains the refund policy, adding retrieval only makes the stale policy easier to find.

OpenAI's research on hallucinations discusses how training and evaluation incentives can reward guessing. In a product, make missing information a supported outcome. The interface should allow a useful clarification or an explicit inability to verify the answer.

Check retrieval and answer generation separately

When using retrieval, inspect whether the relevant authorized passage reached the model. Then check whether the answer accurately reflects it. Those are separate tests. If the answer cites a source, also check that the user can open it and is allowed to see it.

Even an answer based on a source can omit an exception or mix rules from different versions. Stanford's research on legal AI tools found errors in the products it tested despite retrieval. Those results concern that benchmark and those tools, not a universal error rate for AI products.

Retrieval is not always necessary. A small, current set of authorized documents may fit directly in the prompt. A structured database query may be more reliable for a precise account fact. Choose the source path using the task and access requirements.

Separate a suggestion from permission to act

A model may draft a refund response. Code should still enforce eligibility, account ownership, and refund limits before money moves.

Treat retrieved text as evidence, not instructions to change the application's rules. A document saying "ignore the approval requirement" must not bypass a permission check.

Use structured fields when they make an answer easier to validate. A schema can enforce the presence of an order ID and an outcome field. It cannot prove the ID belongs to the customer or the outcome is correct. Those checks need trusted data and application logic.

Give uncertain requests a useful destination

Situation	Product behavior
A required detail is missing	Ask a specific follow-up question
The relevant policy cannot be found	Say what could not be verified
Two applicable sources disagree	Send the conflict for review
The answer would authorize a consequential action	Apply the separate approval policy
Evidence supports a permitted answer	Respond with the relevant source

Avoid adding a confidence threshold just because the API returns a number. Validate that signal on labeled requests from your domain. Count wrong answers that pass the check and answerable questions that are needlessly sent to a person.

A review queue also needs an owner. Preserve the question, evidence, and reason for escalation, with appropriate access and retention controls. Otherwise the system merely transfers an unresolved problem to another screen.

Measure incorrect answers and missed opportunities

Create evaluations that cover ordinary questions, ambiguous requests, exceptions, and unavailable evidence. Check whether the evidence supports each claim that could affect the customer's decision. Also measure whether the feature declines questions it should answer.

Evaluation best practices emphasize task-specific checks and continued evaluation. Review automated scoring against human judgments, especially when the model is grading subtle factual claims.

After launch, connect corrections and complaints to the recorded request, sources, and answer. Track repeated failure categories and feed them back into source maintenance, retrieval, prompts, and model evaluation. Changing all of those at once makes it harder to identify what helped.

Take one recent wrong answer into the next review. Find the first point where the system lacked evidence or ignored it, then add a check at that point. That is a more useful starting task than a general instruction to "make the AI more accurate."

Sources

• OpenAI: Why language models hallucinate
• Stanford HAI: Benchmarking legal AI tools
• OpenAI: Evaluation best practices