Part I — Foundations of Technology Evaluation

Chapter 3. Why Technology Evaluation Is Difficult

3.1 Introduction

If technology evaluation were simply a matter of measuring performance, technology decisions would be relatively straightforward.

However, emerging technologies create difficult evaluation problems.

Evidence may be incomplete.

The technology may be changing.

Users may behave differently from expectations.

Regulations may not yet exist.

Environmental effects may not be fully known.

Costs may change rapidly.

Technological dependencies may emerge.

Stakeholders may disagree about what constitutes success.

Consequently, technology evaluation is fundamentally a problem of decision-making under uncertainty.

3.2 Uncertainty

Uncertainty occurs when decision-makers lack complete knowledge concerning outcomes.

It is useful to distinguish several categories.

Table 3.1 — Categories of Technological Uncertainty

The evaluator should not conceal uncertainty.

Uncertainty is itself an evaluation finding.

3.3 Complexity

Complexity arises when multiple components interact.

For example, the success of an AI healthcare system may depend on:

algorithm quality;

data quality;

clinician behaviour;

patient behaviour;

hospital infrastructure;

interoperability;

regulation;

cybersecurity;

reimbursement;

organizational culture.

Failure in one component can affect the whole system.

Therefore, evaluating components independently may not provide an accurate picture of system-level performance.

3.4 The Problem of Incomplete Data

Emerging technologies frequently lack long-term evidence.

Researchers may have:

laboratory results;

pilot data;

simulations;

expert judgments;

limited operational data.

But they may not have:

ten-year performance data;

long-term maintenance records;

complete environmental assessments;

large population datasets;

mature cost information.

This creates an evidence hierarchy problem.

Table 3.2 — Evidence Strength in Emerging Technology Evaluation

No single evidence type is sufficient in every case.

3.5 The Hype Problem

Emerging technologies are often surrounded by exaggerated expectations.

Technology hype can influence:

investment;

media narratives;

policy;

consumer expectations;

organizational decisions.

The evaluator must distinguish between:

technological capability;

demonstrated performance;

anticipated capability;

commercial claims;

speculative claims.

A disciplined assessment therefore asks:

“What evidence supports this claim?”

rather than:

“How impressive is the claim?”

3.6 The Pilot-to-Scale Problem

A technology can perform successfully in a pilot and fail at scale.

Reasons include:

infrastructure limitations;

increased complexity;

cost escalation;

user resistance;

cybersecurity exposure;

supply constraints;

regulatory barriers;

maintenance problems.

Therefore:

Pilot success ≠ guaranteed large-scale success.

This is one reason why technology readiness and organizational readiness must be assessed separately.

3.7 Path Dependency

Technological development is influenced by previous investments.

Organizations may continue using a technology because they have already invested heavily in:

infrastructure;

training;

software;

data;

contracts;

organizational processes.

This creates path dependency.

A technically superior alternative may therefore face barriers because the existing system is deeply embedded.

Technology evaluation should consequently include transition costs.

3.8 Lock-In

Technology lock-in occurs when users or organizations become dependent on a technological ecosystem.

Examples may include:

proprietary software;

vendor-specific infrastructure;

specialized data formats;

exclusive platforms;

incompatible systems.

Lock-in can create switching costs.

Therefore, technology evaluation should include:

interoperability;

portability;

exit costs;

alternative suppliers;

open standards.

3.9 Unintended Consequences

Technology may produce outcomes that were not part of the original design objective.

An automated system introduced to increase efficiency may increase employee workload because workers must monitor system errors.

A digital platform introduced to improve access may exclude individuals without internet connectivity.

A surveillance system introduced for security may affect privacy.

An AI system introduced to reduce administrative work may introduce new verification requirements.

Technology evaluation should therefore include indirect consequences.

3.10 Black-Box Systems

Some technologies, particularly complex AI systems, may be difficult for users to understand.

The “black-box” problem concerns situations where system outputs can be generated without sufficient transparency regarding the processes producing those outputs.

This creates challenges for:

accountability;

auditing;

explanation;

error correction;

trust.

The problem is not necessarily that every technological process must be fully transparent to every user. Rather, appropriate levels of explainability and accountability should correspond to the consequences of the decision.

A low-risk recommendation system and a system supporting high-stakes decisions may require different governance arrangements.

3.11 Measurement Problems

Technology evaluation requires measurement.

But what should be measured?

Suppose a university introduces an AI tutoring system.

Possible measures include:

number of users;

cost per student;

response time;

student satisfaction;

assessment performance;

retention;

learning outcomes;

staff workload.

These measures may produce different conclusions.

Therefore, evaluators must distinguish:

Activity → Output → Outcome → Impact.

Example

Number of students using AI tool = activity indicator.

Number of interactions = output indicator.

Improved learning = outcome indicator.

Improved educational attainment over time = potential impact indicator.

The distinction prevents organizations from confusing usage with value.

3.12 Stakeholder Conflict

Stakeholders may have different objectives.

Management may prioritize cost reduction.

Employees may prioritize job security.

Customers may prioritize quality.

Regulators may prioritize safety.

Investors may prioritize returns.

Communities may prioritize environmental protection.

Researchers may prioritize knowledge generation.

A technology evaluation must therefore identify whose interests are represented and whose interests may be overlooked.

3.13 Time Horizon

Technology consequences operate across different time periods.

Short-term:

implementation costs;

training;

productivity;

disruption.

Medium-term:

organizational restructuring;

market effects;

workforce changes.

Long-term:

environmental consequences;

social transformation;

technological dependency;

institutional effects.

An evaluation that considers only the first year may produce a different conclusion from one considering ten or twenty years.

3.14 Known Knowns, Known Unknowns and Unknown Unknowns

A practical technology-evaluation framework should distinguish among:

Known knowns

Information supported by strong evidence.

Known unknowns

Questions that have been identified but for which evidence remains incomplete.

Unknown unknowns

Potential issues that have not yet been recognized.

The third category is particularly difficult.

It cannot be directly measured because the evaluator does not yet know what to look for.

This reinforces the importance of:

scenario analysis;

stakeholder diversity;

monitoring;

adaptive governance;

continuous reassessment.

3.15 Risk Versus Uncertainty

Risk and uncertainty are related but conceptually distinct.

Risk generally involves situations in which possible outcomes and their probabilities can be estimated with some basis.

Uncertainty may involve situations where probabilities themselves are poorly known.

For example:

If historical data show a system fails approximately 1% of the time, an organization may be able to estimate failure risk.

If a completely new technology has never been deployed at scale, the probability of certain failures may be impossible to estimate reliably.

This distinction has major implications for decision-making.

3.16 The Evaluation Paradox

A central paradox exists in emerging technology evaluation:

The technology needs evidence to justify adoption, but evidence can often only be generated through adoption.

This creates a dilemma.

A technology may require real-world experimentation to establish effectiveness, yet full deployment may be inappropriate until effectiveness is demonstrated.

The practical solution is often controlled experimentation.

Examples include:

pilot projects;

sandbox environments;

phased deployment;

limited user groups;

independent monitoring;

reversible implementation.

3.17 Case Study: Autonomous Vehicles

Autonomous vehicles demonstrate the evaluation challenge.

Evaluation requires more than measuring driving accuracy.

Relevant questions include:

How does the system behave in unusual weather?

How does it interact with pedestrians?

What happens during sensor failure?

How does responsibility operate after an accident?

How does the system respond to road infrastructure differences?

What happens when human drivers interact with autonomous systems?

What cybersecurity vulnerabilities exist?

What are the employment effects?

What happens to insurance systems?

The technology therefore requires technical, legal, social, ethical and organizational evaluation.

3.18 Evaluation Under Deep Uncertainty

When uncertainty is high, traditional cost-benefit analysis may not be sufficient.

Additional approaches include:

scenario planning;

sensitivity analysis;

real-options thinking;

adaptive management;

pilot studies;

staged investment;

robust decision-making.

The objective is not to calculate the future perfectly.

It is to make decisions that remain defensible across plausible futures.

3.19 Practical Uncertainty Register

Table 3.3 — Technology Uncertainty Register

This type of register should accompany major technology decisions.

3.20 Chapter Research Questions

Why is emerging technology evaluation characterized by uncertainty?

How can organizations distinguish risk from uncertainty?

Why may pilot success fail to translate into large-scale success?

How do stakeholder conflicts influence technology evaluation?

What methods can organizations use when evidence is incomplete?

3.21 Chapter Summary

Technology evaluation is difficult because technologies are dynamic, complex and socially embedded.

The major challenges include uncertainty, incomplete data, technological hype, scale effects, path dependency, lock-in, unintended consequences, measurement problems, stakeholder conflict and longtime horizons.

The appropriate response is not to abandon evaluation because certainty is unavailable.

Instead, evaluators should explicitly document uncertainty, use multiple evidence sources, conduct scenario analysis, involve stakeholders, use pilots where appropriate and continuously monitor outcomes.

Chapter 4 builds on these foundations by presenting the Adams Multidimensional Technology Evaluation Framework.

3.22 Discussion Questions

What is the difference between risk and uncertainty?

Why can a successful pilot fail at scale?

How can technology hype influence decision-making?

What are the dangers of technological lock-in?

How should organizations respond to unknown unknowns?