Myth #6: 99.9% Accuracy.

99.9% Accuracy on What?

You see claims like "99.9% accuracy" in AI pitches all the time. The number may even be correct. The problem is that without knowing what was tested, it tells you very little.

Usually the test is narrow and controlled. Give the model a document and ask it to extract a field. Give it a contract and ask it to summarize the delivery terms. Give it a set of facts and ask questions about them. When the information needed to answer the question is sitting directly in front of the model, accuracy can be very high.

That is not how most operational decisions work.

Can We Split This Shipment?

Your team is more likely to ask: "Can we split this shipment and still meet the customer commitment?" Answering that might require open orders, inventory from several systems, supplier confirmations, production dates and contract terms. The system may also need to understand that two applications define "available" differently, that a supplier confirmation is not always reliable, or that this customer has specific rules around partial deliveries.

The 99.9% measured on a document extraction test tells you almost nothing about performance on that question.

Accuracy drops when information is missing, sources conflict, business rules need to be applied, or the model has to work through several steps before producing an answer. Grounding the model in verified company data helps significantly, but it does not eliminate errors or hallucinations.

Beware the Vendor Who Never Hallucinates.

Today's language models hallucinate. Better models can reduce the problem, and grounding them in verified data can reduce it dramatically, but with today's architectures it cannot be eliminated entirely. The floor is not zero.

So I would be cautious of any vendor saying their system "doesn't hallucinate." The important question is not whether they have somehow eliminated the problem. It is whether they have engineered around it.

A well-designed system assumes errors can happen. It retrieves facts from verified sources rather than relying on the model's memory, makes important claims traceable to the underlying record, recognizes when information is missing or conflicting, and knows when to stop rather than fill a gap with a plausible answer. Human approval should sit around the decisions where getting something wrong could have a material consequence.

So don't ask whether the system hallucinates. Ask what happens when it does.

Twelve Steps, 54% Reliable.

There is another problem with accuracy claims: real business workflows usually involve multiple steps. The system might retrieve data, interpret it, compare it with another source, apply a rule, make a recommendation and then take an action. An error at one step can affect everything that follows.

If a workflow has twelve independent steps and each step is 95% accurate, the probability of all twelve being correct is about 54%. Real workflows are not this simple, and errors are not always independent, but the example shows why per-step accuracy and workflow reliability are not the same thing.

A model can therefore perform very well in isolation while the process built around it is unreliable. What matters is not only the accuracy of an individual model response, but whether the complete workflow produces the right outcome.

Make the Demo Prove It.

So when a vendor shows you an accuracy number, ask three questions.

Accuracy on what task? A result from document summarization or data extraction is only useful if that is what you are actually buying the system to do.

Measured on whose data? A benchmark or vendor test is useful, but I would want to repeat the test using my own data, including incomplete records, conflicting information and the exceptions that occur in the real operation.

What is the end-to-end accuracy? If the system retrieves, interprets, reconciles, recommends and acts, I want to know how reliably that whole process works, not just the accuracy of one step.

I would also change how the demo is run. Give the system a real business question and then inspect the answer. Where did this number come from? Show me the source. Why was this source used? What happens if that information is missing? What happens if two systems disagree? Show me an example where the system refuses to answer because it does not have enough evidence.

Those tests are much closer to how the system will behave once it is running inside your business.

Stress-test AI accuracy on your own data with a free blueclip readiness assessment →