Chapter 12. Measuring Value: Projects, Procurement, and Implementation
12.1 Start with the problem
An AI project should begin with a defined operational or social problem, not a technology demonstration. Describe the current process, who experiences the problem, how often it occurs, what it costs, and why existing approaches are insufficient. Identify the people who will use the system and those who may be affected by it.
Define success before choosing a tool. Measures may include task completion time, error rate, quality, accessibility, user satisfaction, cost, service outcomes, risk reduction, or staff capacity. Include safeguards so that a speed improvement does not come at the expense of quality or fairness.
12.2 Business case and total cost
An AI business case should include licence or usage fees, implementation, data preparation, integration, training, governance, evaluation, cybersecurity, maintenance, monitoring, and exit costs. Also consider costs from errors, delays, vendor dependence, and the need for human review.
Benefits should be realistic. Time saved is not automatically cash saved. A productivity gain becomes economically meaningful when it improves throughput, quality, service, capacity, or cost in a way the organisation can measure.
12.3 Pilot design
A pilot should have a limited scope, defined users, approved data, success criteria, a baseline, a time limit, and a decision point. Compare performance with the current process, and include difficult examples rather than only easy demonstrations. Record both positive outcomes and failures.
Before the pilot begins, decide what would lead to stopping, changing, or scaling the project. This prevents a project from continuing simply because time and money have already been spent.
12.4 Procurement questions
Ask the provider about data handling, model updates, retention, training use, security certifications, access controls, audit logs, uptime, support, accessibility, evaluation evidence, intellectual-property terms, incident notification, data export, portability, and termination.
Clarify who is responsible when the system gives an incorrect result or causes harm. A contract cannot remove the organisation's own duties to customers, staff, students, or the public.
12.5 Evaluation framework
Use a balanced scorecard. Operational measures show speed and throughput; quality measures show accuracy and error rates; human measures show trust, workload, and usability; risk measures show incidents and policy compliance; financial measures show total cost and realised benefit; equity measures show whether performance differs across groups.
Review the system after launch. Monitor changes in data, user behaviour, model versions, costs, and error patterns. A system that performed well during a short pilot may fail under larger scale, new populations, or changed conditions.
12.6 Example: customer-service assistant
Suppose a business wants to help staff answer routine questions. A safe pilot could restrict the assistant to approved public product information and internal non-sensitive procedures. It should draft answers for staff review rather than automatically sending them at first.
Baseline measures could include average handling time, first-contact resolution, error rate, escalation rate, customer satisfaction, and staff workload. The pilot should test confusing questions, outdated documents, unsupported requests, and attempts to obtain private information. Scale only if measured benefits exceed costs and quality and privacy safeguards work.