Measuring an AI Deployment: What to Agree Before You Build

"Is the AI working?" sounds like a simple question. Without agreed measures it becomes a matter of opinion — and opinions tend to follow whoever is most enthusiastic or most sceptical in the room. The time to settle it is before anything is built.

Measure the workflow, not the model

Technical accuracy scores are useful to engineers, but they rarely answer the business question. Focus instead on what happens to the work itself.

Useful measures usually fall into four groups:

• Speed — time from request received to output delivered. For example, RFQ received to quote sent.
• Effort — people-time spent per case, including review.
• Quality — the share of AI outputs approved without significant edits, and the kinds of edits people make.
• Coverage — the share of cases the AI handles versus escalates to a person.

Two or three of these, chosen for the specific workflow, are usually enough.

Capture a baseline first

Measures only mean something against a starting point. Before the AI goes live, record how the workflow performs today: how long cases take, how many are handled per week, where delays happen. Even a rough baseline from a few weeks of observation is far better than none.

Decide what good looks like

Agree in advance what result would justify expanding the deployment, what would call for adjustment, and what would mean stopping. Writing this down protects everyone from moving the goalposts later.

Watch the edits

The edits people make to AI output are one of the most valuable signals you have. If reviewers keep correcting the same thing — a pricing rule, an item mapping, a tone of voice — that tells you exactly what to improve. Logging edits by type turns review from a chore into a feedback loop.

Look beyond time saved

Time is the obvious benefit, but not the only one. Faster responses to customers, more consistent handling, fewer missed follow-ups and better visibility for managers all matter. Include the ones that matter to the business owner, and measure them as directly as possible.

Review on a schedule

Set regular reviews — for example after the first few weeks, and then monthly — where the business owner and the team look at the measures together. That rhythm keeps the deployment honest, and makes decisions about extending AI to the next workflow straightforward.