The measurement problem starts before the build
Most AI projects cannot prove whether they worked, and the reason is almost never the analysis. It is that nobody captured a baseline.
Six months after go-live someone asks what the return was. The team reconstructs it: the process "used to take about two hours" and "now takes about twenty minutes". Both figures are estimates produced by people with an interest in the answer, compared against each other, and reported as a result. Everyone involved knows it is soft. Nobody says so, because the alternative is admitting the project cannot be evaluated.
The fix is unglamorous and it has to happen first: agree one metric before the build starts, measure it for two weeks while the process is still manual, and instrument the system to report the same metric afterwards. Two weeks of slightly tedious data collection is the difference between reporting a result and asserting one.
One metric, not a dashboard of twelve. A dashboard is what you build when you are not confident which number matters.
Why hours saved flatters
"Hours saved" is the default metric because it is the easiest to produce and the easiest to make large. It is also the easiest to misread, in three specific ways.
Hours are not money until something fills them. If a system returns six hours a week to a team of four and nothing changes about what that team does, you have not saved money. You have created slack. Slack has real value, since it absorbs peaks, reduces errors and makes people less miserable, but it is not a cost saving and reporting it as one will not survive contact with a CFO.
Hours become money in exactly three ways: headcount you did not have to add, overtime or contractor spend you stopped, or revenue-generating work that now fits into the same week. Say which one you are claiming.
Aggregated small savings across many people rarely materialise. Ten minutes a day saved for thirty people is 150 hours a month on a slide and close to nothing in practice, because ten minutes does not aggregate into deployable capacity. It disperses. Concentrated savings, such as a whole role's worth of work removed from two people, behave completely differently, and are the ones that show up in a budget.
The comparison is usually unfair. Manual timings get measured on a normal day. Automated timings get measured on the happy path, excluding the exception queue, the reviews and the reruns. Count the whole system, exceptions included, or you are comparing two different processes.
Five metrics that hold up
Pick the one that matches the actual business constraint. If you cannot say which constraint the project is relieving, that is worth resolving before spending money.
| Metric | Use when | Measure as |
|---|---|---|
| Cycle time | Speed of response is the commercial constraint | Median and 90th percentile hours from arrival to completion |
| Cost to serve | Margin per transaction is under pressure | Fully loaded cost per item, including exceptions and review |
| Throughput per head | You want growth without proportional hiring | Items completed per FTE per week |
| Exception rate | Quality and rework are the real cost | Percentage requiring human intervention, and rework hours |
| Capacity released | The team is the bottleneck and you know what fills it | Hours returned, plus a named use for them |
Two notes on using these.
Report medians and 90th percentiles, never averages alone. Averages hide the tail, and the tail is where the customer experience lives. A process with a two-hour median and a six-day 90th percentile has a serious problem that a "twelve-hour average" conceals entirely.
And for capacity released, the named use is not optional. "Six hours a week returned to the credit team, which absorbed the volume growth we would otherwise have hired for" is a claim. "Six hours a week returned" is a number.
Capturing a baseline you can defend
Two weeks, while the process is still manual. Four things:
- Volume. Items per day, with the daily and weekly shape. Peaks matter more than the average, because peaks are usually where the pain is.
- Time per item. Sampled honestly, with start and stop timestamps on real items, not a manager's estimate. Include the interruptions; they are part of the real cost.
- Error and rework rate. How often does something come back? Almost nobody measures this beforehand, and it is often where the largest saving turns out to be.
- Elapsed time, not just handling time. How long an item sits in a queue before someone touches it. In most processes the waiting dwarfs the working, and automation's biggest effect is on the waiting.
The last one is consistently underestimated. A task that takes four minutes of work and three days of sitting in a queue has a three-day cycle time, and fixing that is worth far more to a customer than the four minutes.
Counting the whole cost
Returns are usually stated honestly and costs almost never are. A complete picture includes:
- Build. The one-off, including the discovery that made it scopeable.
- Model usage. Per run, times volume, projected at your expected growth rather than today's.
- Infrastructure. Hosting, storage, monitoring.
- Maintenance. Evaluation runs, prompt and rule updates, adapting to a vendor's API change. Budget something real; systems that touch other systems need attention.
- Human review. The gate is a running cost. Ten items a day at two minutes each is a real number and it belongs in the model.
- Exception handling. The items that fall out still need working, sometimes more expensively than before because they are now the hard ones.
That last point is worth dwelling on. Automation takes the easy items first, which means the residual manual work is disproportionately the difficult work. Per-item manual cost typically rises after automation, even as total cost falls. If your model assumes the leftover items cost what the average item used to, it is wrong in a direction that will be discovered later.
Reading the result honestly
A few tests that separate a real result from a flattering one:
- Would this number survive someone hostile checking it? If the answer depends on excluding the exception queue, it is not a result.
- Is the comparison like for like, with the same period length, seasonality and definition of done?
- Has anything else changed at the same time? A new hire, a system upgrade, a quiet quarter. If so, say so.
- Is the claimed saving visible anywhere in a budget line? If it is not, it is capacity, and capacity should be called capacity.
Reporting a smaller number you can defend is worth more than a larger one you cannot, because the second project's funding depends on the first project's numbers being believed.
What to do when the number disappoints
Sometimes the honest measurement says the return was modest. Three things are worth doing before concluding the approach was wrong.
Check whether the constraint moved. Frequently the bottleneck was not where everyone assumed, and automating the assumed bottleneck relieved nothing. That is a useful finding, because you now know where the real one is.
Check the exception rate. A system escalating forty per cent of items is doing far less work than one escalating five. Often the fix is not more capability but a narrower scope: automate the eighty per cent of cases that are clean and stop trying to handle the rest.
Check whether the capacity went anywhere. If the hours came back and dispersed, the problem is not the system but that nobody decided what the capacity was for. That is a management decision, and it should have been made before the build.
And if none of those explain it, report that honestly too. A consultancy that only ever reports wins is a consultancy whose numbers you cannot use. The value of measuring properly is that occasionally it tells you something you did not want to hear, which is precisely when it is worth the most.