Is Your AI Pilot Actually Helping? Measure Before You Expand

Your AI trial produces a polished draft in seconds. The next question is harder: does it help your team finish useful work, at an acceptable cost and standard?

For business owners and operations leaders, a convincing demonstration is a starting point. Before buying more licences or connecting more systems, compare the whole workflow with how the job is done today. Include the checking, corrections and exceptions that sit between an AI response and a result someone can use.

Agree what a finished job looks like

Choose a task with a clear end point. “Help with customer enquiries” is too broad to measure. “Prepare a draft response to a routine delivery enquiry for a staff member to check” gives the team something concrete to test.

For that example, a usable draft might need the correct order reference, delivery status supported by the approved source, and a clear next step. It must avoid inventing a delivery promise. Agree these requirements with the person responsible for the work before judging the outputs. This is an illustrative test, not a claim about a Zarbtech customer deployment.

The National AI Centre’s AI adoption guidance recommends acceptance criteria that reflect the intended use and risks, documented testing, and approval based on the results. Turn that principle into a short definition of “ready for a person to use” for your chosen task.

Measure the current process first

Watch a representative mix of work using the current process. Record hands-on time, waiting time, corrections and the outcome. Keep these measures separate: a faster draft may not shorten a customer’s wait if approval remains the bottleneck.

Use comparable cases for the AI trial. If the existing process handles difficult enquiries while the pilot receives only tidy examples, the comparison will flatter the pilot. Include ordinary variation, such as an incomplete request, an outdated document or two records that disagree. Use information approved for the trial and the selected tool.

Count the work after the first answer

Have reviewers record the time spent preparing inputs, checking sources, editing the response and handling failures. A useful working measure is total staff effort per accepted result. Also note cases abandoned and completed manually; leaving them out hides the cost of unsuccessful attempts.

Suppose a draft arrives quickly but a reviewer must reconstruct its reasoning from several systems. That may still be useful for some tasks, but the business case needs to include that effort. Ask reviewers which parts they can verify confidently and which require specialist knowledge. Their ability to check the work is part of the design.

Separate minor edits from serious failures

An awkward sentence and a wrong customer record should not count as equivalent errors. Group findings by consequence: presentation fixes, factual corrections, missing information and failures that could expose information or trigger an inappropriate action. Agree which findings require the trial to pause, regardless of its average score.

For the delivery example, test whether the draft acknowledges an unknown delivery date rather than guessing. Check that it uses the correct customer’s information. Keep staff approval in place while testing; a pilot that drafts a response does not establish that it is ready to send responses automatically.

The Digital Transformation Agency’s reliability and safety guidance explains why testing needs varied scenarios and why accuracy must be assessed in context. It is guidance for government agencies, rather than a blanket requirement for private businesses, but those testing principles are useful when designing a business pilot.

Put the costs beside the results

List recurring licences or usage charges separately from setup, integration, training and ongoing support. Include the effort needed to keep source material current. Compare these with the volume of work the team can realistically use the tool for, rather than assuming every employee will use every feature.

Time released is not automatically a cash saving. Decide what the team would do with that capacity: clear a backlog, improve response quality or absorb more work. Name the intended benefit and how you will observe it. Avoid counting the same improvement twice as both time saved and reduced staffing cost.

Make a decision you can revisit

Bring a one-page review to the decision meeting: task and sample tested, accepted results, review effort, significant failures, expected operating costs and unresolved issues. Choose whether to expand within the tested scope, adjust and retest, or stop. A small sample can identify problems; it cannot prove that uncommon failures will never occur.

If you proceed, assign an owner and a review date. Retest when the model, instructions, source documents or connected systems change in ways that could affect the result. Keep a practical route back to the existing process if performance falls.

Build your pilot around evidence

Zarbtech’s approach is to connect AI investment to useful work and measurable outcomes. Our Managed AI service includes workflow assessment, defined success measures, pilot planning and ongoing performance and cost review. Bring one task, its current process and the result you want to improve. We can help scope a pilot and the evidence needed for your next decision.