Part II — Dimensions of Technology Evaluation

Chapter 5. Technical and Functional Evaluation of Emerging Technologies

5.1 Introduction

The first question that organizations commonly ask about a technology is deceptively simple:

“Does it work?”

Although this question is essential, it is not sufficient.

A technology may operate according to its technical specifications while failing to solve the problem for which it was introduced. Conversely, a technology may demonstrate strong potential but require additional development before it can operate reliably in a real-world environment.

Technical evaluation therefore needs to be separated from functional evaluation.

Technical evaluation asks whether the technology performs according to specified technical requirements.

Functional evaluation asks whether the technology achieves the intended purpose in its actual context of use.

This distinction is fundamental to the Adams Multidimensional Technology Evaluation Framework (AMTEF).

A technology that is technically sophisticated but functionally ineffective cannot be regarded as successful.

Similarly, a technology that performs well under laboratory conditions but fails when exposed to real-world complexity requires further evaluation before widespread deployment.

The technical dimension of AMTEF therefore concerns performance, reliability, accuracy, scalability, interoperability, maintainability, robustness, resilience and maturity. The functional dimension concerns usefulness, problem-solving capability, user outcomes, effectiveness and fitness for purpose.

Together, these dimensions establish the technological foundation upon which economic, organizational, social, ethical and environmental assessments can be constructed.

5.2 Technical Evaluation: The Basic Question

Technical evaluation examines the characteristics of the technological system itself.

Depending on the technology, these may include:

accuracy;

precision;

speed;

capacity;

reliability;

availability;

durability;

scalability;

interoperability;

maintainability;

robustness;

resilience;

security;

energy efficiency;

resource requirements;

fault tolerance;

technological maturity.

The relevant measures differ according to the technology.

For example, evaluating a medical diagnostic AI system may require sensitivity and specificity.

Evaluating a battery may require energy density, cycle life, charging speed and degradation.

Evaluating a cloud platform may require availability, latency, scalability and cybersecurity.

Evaluating a robot may require movement accuracy, safety, reliability and task completion.

Consequently, technical evaluation must begin by defining the technological function.

5.3 Technical Requirements

Before performance can be measured, requirements must be established.

A technical requirement should describe what the system must be capable of doing.

For example:

“The system shall process 10,000 transactions per minute.”

“The diagnostic system shall achieve a specified level of sensitivity under defined clinical conditions.”

“The battery shall maintain a specified percentage of capacity after a defined number of charge cycles.”

The importance of requirements is that they establish a basis for objective assessment.

Without defined requirements, evaluators may confuse impressive performance with adequate performance.

5.4 Functional Requirements and Non-Functional Requirements

Technology requirements can be divided into two broad categories.

Functional requirements

These describe what the system must do.

Examples:

process transactions;

classify images;

generate reports;

control machinery;

store information;

communicate with other systems.

Non-functional requirements

These describe how the system must perform.

Examples:

reliability;

security;

availability;

scalability;

usability;

maintainability;

response time.

Table 5.1 — Functional and Non-Functional Requirements

5.5 Accuracy

Accuracy is one of the most commonly used technical measures.

However, accuracy must be interpreted carefully.

Suppose an AI system classifies 95% of cases correctly.

Is 95% acceptable?

The answer depends on context.

For a low-risk recommendation system, it may be acceptable.

For a high-stakes medical diagnostic application, it may not be.

Accuracy can also be misleading where datasets are imbalanced.

For example, if 95% of cases belong to one category, a system that simply predicts the dominant category could achieve 95% accuracy while providing little practical value.

Therefore, technical evaluation should use context-appropriate metrics.

5.6 Precision, Recall and F-Measure

For classification systems, particularly AI systems, several measures may be relevant.

Precision asks:

Of the cases predicted as positive, how many were actually positive?

Recall asks:

Of the cases that were actually positive, how many were identified?

The F1 score provides a harmonic balance between precision and recall.

These measures demonstrate a broader principle:

Technical evaluation should match the consequences of errors.

False positives and false negatives do not necessarily have equal consequences.

5.7 Reliability

Reliability concerns the probability that a system performs its required function under specified conditions for a specified period.

Reliability is particularly important for:

industrial systems;

healthcare technologies;

transportation;

telecommunications;

energy infrastructure;

financial systems.

A technology that works successfully during demonstrations but fails frequently during continuous operation has limited practical value.

Reliability should therefore be assessed over time rather than through one-off demonstrations.

5.8 Availability

Availability refers broadly to whether a system is accessible and operational when required.

A system may be technically functional but unavailable because of:

network failure;

power interruption;

server failure;

software updates;

cybersecurity incidents;

maintenance.

Availability is especially important for critical infrastructure.

An emergency healthcare system, for example, cannot be evaluated solely according to functionality when the system is available. Its performance during outages and disruptions is also relevant.

5.9 Robustness

Robustness concerns the ability of a technology to continue functioning under variation or imperfect conditions.

Examples include:

noisy data;

changing environments;

unusual inputs;

hardware variation;

incomplete information;

unexpected user behaviour.

A robust AI system should not fail dramatically when inputs differ slightly from training conditions.

A robust industrial system should tolerate reasonable variation in operating conditions.

5.10 Resilience

Robustness and resilience are related but different.

Robustness concerns continued performance under disturbance.

Resilience concerns the capacity to absorb disruption, recover and continue operating.

A resilient system may:

detect failure;

isolate the problem;

switch to an alternative;

recover;

restore normal operation.

This is increasingly important in interconnected technological ecosystems.

5.11 Scalability

Scalability concerns whether a technology can expand without unacceptable degradation in performance or cost.

A system may perform exceptionally well with:

100 users;

but poorly with:

100,000 users.

Similarly, a manufacturing process may work at laboratory scale but become difficult or expensive at commercial scale.

This is particularly important for emerging technologies.

Moni et al. (2020) identify scale-up uncertainty as an important challenge in assessing emerging technologies through lifecycle analysis.

5.12 Interoperability

Interoperability is the ability of different systems to communicate and work together.

It is increasingly important because organizations rarely use a single technological system.

A hospital may have:

electronic health records;

laboratory systems;

imaging systems;

pharmacy systems;

AI applications.

If these systems cannot exchange information effectively, the technological value of each individual system may be reduced.

Interoperability should therefore be evaluated through:

data formats;

APIs;

standards;

communication protocols;

semantic compatibility;

identity management.

5.13 Maintainability

Maintainability concerns how easily a system can be repaired, updated and modified.

A technology with low initial cost may become expensive if:

updates are difficult;

spare parts are unavailable;

specialist engineers are scarce;

documentation is poor;

software architecture is proprietary.

Technology evaluation should therefore consider lifecycle maintenance rather than initial acquisition alone.

5.14 Usability

Usability concerns whether intended users can effectively interact with the technology.

A technically sophisticated technology may fail because users find it:

confusing;

time-consuming;

difficult to learn;

inaccessible;

poorly designed.

Human-computer interaction therefore forms part of technical and functional evaluation.

ISO 9241-11 conceptualizes usability in relation to effectiveness, efficiency and satisfaction within a specified context of use (International Organization for Standardization, 2018).

5.15 Safety

Safety is a critical technical criterion for technologies whose failure can cause physical or serious societal harm.

Relevant technologies include:

medical devices;

autonomous vehicles;

industrial robots;

aviation systems;

energy infrastructure;

chemical technologies;

biotechnology.

Safety evaluation should consider:

normal operation;

abnormal operation;

failure conditions;

human error;

system interaction;

maintenance;

emergency conditions.

5.16 Cybersecurity as a Technical Requirement

Digital technologies increasingly require cybersecurity evaluation.

Technical assessment should examine:

authentication;

authorization;

encryption;

vulnerability;

intrusion detection;

data integrity;

software updates;

supply-chain security.

Cybersecurity cannot be treated solely as a separate administrative function.

It is part of technical system performance.

5.17 Technology Readiness Levels Revisited

TRLs provide a useful maturity framework.

Table 5.2 — TRL and Evaluation Implications

However, a high TRL does not automatically establish economic, organizational or social readiness.

AMTEF therefore treats TRL as one input into technical evaluation rather than a complete adoption criterion.

5.18 Functional Effectiveness

Functional evaluation begins with the problem.

The evaluator should ask:

“What outcome is the technology supposed to produce?”

This question is essential because organizations sometimes begin with a technology and then search for a problem it might solve.

A more disciplined sequence is:

Problem → Desired outcome → Evaluation criteria → Technology alternatives.

5.19 Problem-Solution Fit

A technology should be evaluated according to whether it addresses the underlying problem rather than merely its symptoms.

For example, suppose an organization has low employee productivity.

Introducing an AI monitoring system may increase surveillance, but it may not address:

poor training;

inadequate staffing;

inefficient processes;

unclear responsibilities;

poor management.

Technology should therefore not be treated as a universal solution.

5.20 Output, Outcome and Impact

Functional evaluation should distinguish three levels.

Output

What the technology produces.

Outcome

What changes because of the output.

Impact

The longer-term consequence.

Table 5.3 — Technology Results Chain

This framework prevents technology use from being confused with technological effectiveness.

5.21 Effectiveness Versus Efficiency

Effectiveness asks:

Did we achieve the intended outcome?

Efficiency asks:

How many resources were required to achieve it?

A technology can be:

effective but inefficient;

efficient but ineffective;

both effective and efficient;

neither effective nor efficient.

Both dimensions should therefore be assessed.

5.22 Comparative Evaluation

Technology should rarely be evaluated in isolation.

The relevant alternatives may include:

current practice;

technology A;

technology B;

hybrid solution;

no intervention.

Table 5.4 — Comparative Technology Evaluation

The table does not determine the final decision. It structures comparison.

5.23 Case Study: AI Diagnostic System

Consider an AI system designed to identify abnormalities in medical images.

Technical evaluation

sensitivity;

specificity;

false-positive rate;

false-negative rate;

processing time;

reliability.

Functional evaluation

Does diagnosis improve?

Does clinician workload change?

Does time to diagnosis improve?

Does patient outcome improve?

Organizational evaluation

Can clinicians use the system?

Is infrastructure sufficient?

Is training available?

Governance evaluation

Who is accountable for incorrect decisions?

Is human review required?

How is patient data protected?

This example demonstrates why technical accuracy is only one part of evaluation.

5.24 Technical Evaluation Protocol

A practical technical evaluation can follow ten steps:

define technical requirements;

define functional requirements;

establish performance indicators;

identify test conditions;

identify baseline performance;

test under normal conditions;

test under abnormal conditions;

assess scalability;

assess interoperability;

document limitations.

5.25 Technical Evaluation Matrix

Table 5.5 — AMTEF Technical Evaluation Matrix

5.26 Chapter Research Questions

How should technical performance be measured across different technology categories?

How can technical readiness be distinguished from functional effectiveness?

How should technology evaluation address scalability?

What role does interoperability play in technology value?

How should usability be incorporated into technical evaluation?

How can organizations test emerging technologies under realistic conditions?

5.27 Chapter Summary

Technical evaluation determines whether a technology performs according to defined requirements.

Functional evaluation determines whether it actually solves the intended problem.

Important technical dimensions include accuracy, reliability, availability, scalability, robustness, resilience, interoperability, maintainability, usability, safety and cybersecurity.

However, no technical measure should be interpreted independently of context.

The central principle is:

A technology that works is not necessarily a technology that works for the intended purpose.

5.28 Discussion Questions

Why is accuracy not always sufficient?

What is the difference between robustness and resilience?

Why is scalability difficult for emerging technologies?

How does interoperability affect technology value?

Why should technology be evaluated against alternatives?

How can a technology be technically successful but functionally unsuccessful?