Part I — Foundations of Technology Evaluation

Chapter 4. The Adams Multidimensional Technology Evaluation Framework

4.1 Introduction

The preceding chapters established three fundamental propositions.

First, emerging technologies are complex sociotechnical systems.

Second, technology assessment has evolved from forecasting and impact assessment toward more participatory, constructive, anticipatory and continuous approaches.

Third, technological evaluation is inherently uncertain.

The logical consequence is that technology evaluation requires a comprehensive framework.

This chapter introduces the Adams Multidimensional Technology Evaluation Framework (AMTEF).

The framework is designed to provide a structured architecture for evaluating technologies across ten dimensions.

The framework does not assume that every dimension has equal importance in every situation.

Instead, it requires evaluators to consider all ten dimensions and then determine their relative relevance according to the technology, context, stakeholders and decision.

4.2 The Ten Dimensions of AMTEF

The ten dimensions are:

Technical Performance

Functional Effectiveness

Economic Value

Organizational Readiness

Human Impact

Social Consequences

Ethical Responsibility

Environmental Sustainability

Risk and Security

Governance and Regulatory Compatibility

Figure 4.1 — ADAMS MULTIDIMENSIONAL TECHNOLOGY EVALUATION FRAMEWORK

Conceptual Framework

▼

▼

▼

▼

▼

▼

▼

▼

▼

▼

The model is deliberately circular.

Technology evaluation does not terminate with implementation; monitoring and reassessment feed the next evaluation cycle.

Figure X.X. Adams Multidimensional Technology Evaluation Framework.

Note. The framework integrates technical, economic, human, ethical, environmental, risk, security, governance, and regulatory dimensions into a continuous decision process. Following evaluation synthesis, an organization may adopt, pilot, or adapt the technology. The resulting implementation is subsequently monitored and reassessed, allowing evidence from one cycle to inform the next.

4.3 Dimension 1: Technical Performance

Technical performance concerns whether the technology performs according to defined technical requirements.

Indicators may include:

accuracy;

speed;

reliability;

scalability;

availability;

interoperability;

robustness;

resilience;

maintainability;

security;

energy efficiency;

performance under stress.

Technical evaluation should use objective evidence where possible.

For software, relevant metrics may include:

system availability;

processing speed;

error rates;

latency;

accuracy;

uptime.

For physical technologies, evaluation may include:

durability;

capacity;

failure rates;

efficiency;

operating tolerance.

4.4 Dimension 2: Functional Effectiveness

Technical performance is not identical to functional effectiveness.

A system can function exactly as designed but fail to solve the user's actual problem.

For example, an organization may deploy a sophisticated analytics system that produces accurate dashboards but does not improve managerial decisions.

Functional effectiveness therefore asks:

Does the technology achieve its intended purpose?

Table 4.1 — Technical Performance Versus Functional Effectiveness

This distinction should become a standard element of technology evaluation.

4.5 Dimension 3: Economic Value

Economic evaluation examines whether expected benefits justify the resources required.

Potential indicators include:

capital expenditure;

operating expenditure;

total cost of ownership;

return on investment;

net present value;

payback period;

productivity;

revenue;

cost reduction;

opportunity cost.

However, economic value should not be reduced to financial return.

A public-health technology may have limited direct financial revenue but generate significant social value.

Therefore, the framework distinguishes:

Financial value → Economic value → Social value → Strategic value.

4.6 Dimension 4: Organizational Readiness

Technology does not implement itself.

Organizations require:

leadership;

skills;

infrastructure;

funding;

policies;

data;

change-management capacity;

technical support;

organizational culture.

Organizational readiness can be assessed through a maturity model.

Table 4.2 — Organizational Readiness Levels

A technology should not be implemented solely because the technology itself is mature.

The organization must also be ready.

4.7 Dimension 5: Human Impact

Human impact examines how technology changes human experience and agency.

Questions include:

Does the technology increase human capability?

Does it replace human tasks?

Does it change professional roles?

Does it increase monitoring?

Does it improve accessibility?

Does it create new burdens?

Does it preserve meaningful human control?

Human-centred evaluation is particularly important when technologies interact directly with people.

4.8 Dimension 6: Social Consequences

Technology can redistribute benefits and costs.

An innovation may improve access for some populations while excluding others.

Therefore, evaluators should examine:

equality;

inclusion;

affordability;

accessibility;

digital divide;

geographical distribution;

social cohesion;

community impacts.

The question should not simply be:

“Does society benefit?”

It should also be:

“Which groups benefit, which groups bear costs, and under what conditions?”

4.9 Dimension 7: Ethical Responsibility

Ethical assessment considers values and responsibilities.

Key principles may include:

dignity;

autonomy;

fairness;

privacy;

transparency;

accountability;

non-discrimination;

human oversight.

For AI systems, these considerations are increasingly important. International guidance has emphasized human rights, dignity, transparency, fairness, human oversight and environmental responsibility in AI governance.

The evaluator should identify both explicit ethical risks and ethical issues embedded in system design.

4.10 Dimension 8: Environmental Sustainability

Environmental assessment should use lifecycle thinking.

The evaluator should consider:

Raw materials → manufacturing → transport → deployment → operation → maintenance → reuse → recycling/disposal.

Relevant indicators include:

greenhouse gas emissions;

energy consumption;

water consumption;

material intensity;

mineral requirements;

pollution;

biodiversity;

waste.

Life-cycle assessment is especially valuable for emerging technologies because environmental impacts can shift across different stages of the lifecycle.

Moni et al. (2020) emphasize that emerging technologies present specific LCA challenges because of limited data, uncertain functionality and scale-up uncertainty. They recommend combining lifecycle assessment with other analytical approaches rather than treating LCA as a standalone answer.

4.11 Dimension 9: Risk and Security

Risk evaluation asks:

“What can go wrong?”

Security asks:

“How can the system be protected against threats?”

Risk categories may include:

technical;

operational;

financial;

cybersecurity;

privacy;

legal;

environmental;

reputational;

systemic.

A basic risk expression can be represented as:

Risk = Probability × Consequence

However, this formula should not be interpreted as universally sufficient.

When probabilities are uncertain, numerical risk scores may create false precision.

Therefore, qualitative risk assessment and scenario analysis may sometimes be more appropriate.

4.12 Dimension 10: Governance and Regulatory Compatibility

The final dimension concerns whether the technology can be governed.

Governance includes:

policies;

laws;

standards;

accountability;

audits;

documentation;

monitoring;

transparency;

human oversight;

complaint mechanisms;

responsibility allocation.

A technically effective technology may still be inappropriate if governance arrangements are inadequate.

4.13 The AMTEF Evaluation Cycle

The framework can be implemented through ten stages.

Stage 1: Identify the Technology

Define precisely what is being evaluated.

Stage 2: Define the Problem

Identify the problem independently of the proposed technology.

This prevents technology-driven decision-making.

Stage 3: Identify Stakeholders

Identify those who:

use;

fund;

regulate;

develop;

manage;

benefit from;

or may be affected by the technology.

Stage 4: Establish Evaluation Criteria

Define measurable criteria across the ten dimensions.

Stage 5: Collect Evidence

Use multiple sources.

Stage 6: Evaluate Uncertainty

Identify known, uncertain and unknown areas.

Stage 7: Assess Alternatives

The relevant comparison is not always technology versus no technology.

It may be:

Technology A versus Technology B versus existing practice.

Stage 8: Pilot

Where appropriate, conduct controlled implementation.

Stage 9: Monitor

Measure actual outcomes.

Stage 10: Reassess

Update the evaluation as evidence changes.

4.14 AMTEF Evaluation Matrix

Table 4.3 — Core Technology Evaluation Matrix

4.15 Weighting the Dimensions

Not every technology requires equal weighting.

A hospital AI system might place greater emphasis on:

safety;

human impact;

ethics;

privacy;

regulatory compliance.

An industrial energy system might place greater emphasis on:

technical performance;

lifecycle environmental impact;

cost;

reliability;

infrastructure.

An educational technology might emphasize:

learning effectiveness;

accessibility;

privacy;

academic integrity;

student outcomes.

However, weighting should not be used to eliminate inconvenient dimensions.

A dimension can receive a low numerical weight while still being subject to minimum requirements.

4.16 Thresholds and Non-Negotiable Criteria

Some technology decisions should use threshold criteria.

For example:

A technology may be economically attractive but fail a mandatory safety requirement.

Therefore:

Overall evaluation = weighted criteria + mandatory thresholds.

This approach prevents a strong performance in one dimension from compensating for an unacceptable failure in another.

Example

CONCEPTUAL FRAMEWORK

AI Healthcare Technology Evaluation with a Patient-Safety Gate

CORE PRINCIPLE

A high weighted average across technical, economic, organizational and environmental dimensions does not override an unacceptable patient-safety result. Patient safety functions as a critical decision gate rather than merely another score in the weighted average.

ILLUSTRATIVE ASSESSMENT

Decision implication: The system should not be automatically approved solely because the weighted average is high.

4.17 Evidence Quality

AMTEF should distinguish between evidence and claims.

Table 4.4 — Evidence Classification

The levels should not be interpreted as universally applicable rankings.

The appropriate evidence depends on the evaluation question.

For example, expert opinion may be highly valuable when assessing emerging strategic technologies for which empirical data are unavailable.

4.18 Stakeholder Mapping

AMTEF recommends stakeholder analysis before evaluation.

Table 4.5 — Stakeholder Mapping Template

Stakeholder analysis helps identify perspectives that might otherwise be excluded.

4.19 Case Study: AI-Assisted University Assessment

Consider a university considering an AI-assisted assessment platform.

Technical

Can the system accurately identify patterns and provide feedback?

Functional

Does feedback improve student learning?

Economic

Does the system reduce administrative cost without creating new costs?

Organizational

Can lecturers understand, monitor and use the system?

Human

Does it strengthen or weaken student agency?

Social

Do students with different levels of digital access receive comparable benefits?

Ethical

How are privacy, fairness, authorship and academic integrity addressed?

Environmental

What computing resources are required?

Risk

What happens if the system produces incorrect or misleading feedback?

Governance

Who is accountable when an AI-generated assessment decision is wrong?

This example demonstrates how AMTEF transforms a general technology question into a structured evaluation.

4.20 AMTEF and Research Design

The framework can support qualitative, quantitative and mixed-methods research.

Quantitative research

Possible variables include:

technology performance;

adoption intention;

organizational readiness;

perceived value;

user satisfaction;

risk perception.

Qualitative research

Possible methods include:

interviews;

focus groups;

observations;

document analysis;

case studies;

expert panels.

Mixed methods

Mixed-methods research can combine:

Survey → Statistical analysis → Interviews → Integration → Evaluation model.

This may be particularly valuable when both measurable outcomes and stakeholder experiences are important.

4.21 Example Research Model

Example Research Model

AI Adoption in Higher Education

Conceptual Framework

1. Organizational Readiness
2. Technology Acceptance
3. Technology Use
4. Educational Outcomes

Moderating / Contextual Influences

Influence the strength or direction of relationships within the model

Model logic: The model proposes that organizational readiness contributes to technology acceptance, which in turn supports actual technology use and ultimately influences educational outcomes. Individual, institutional, and environmental/contextual factors may moderate or condition these relationships.

with moderating or contextual influences from:

ethics;

privacy;

institutional policy;

staff capability;

student digital literacy.

The framework should be adapted to the research question rather than mechanically imposed.

4.22 AMTEF and Decision Categories

The framework generates several possible decisions.

Adopt

Evidence is sufficiently strong and conditions are appropriate.

Pilot

The potential is significant but evidence is incomplete.

Adapt

The technology requires modification or safeguards.

Delay

Evidence or readiness is currently insufficient.

Monitor

The technology is not being implemented but remains strategically significant.

Reject

The technology fails essential requirements within the defined context.

The framework does not determine the decision automatically.

It structures the evidence on which the decision is based.

4.23 Technology Evaluation Scorecard

A scorecard may be useful where numerical evaluation is appropriate.

Table 4.6 — Illustrative AMTEF Scorecard

The weights in this table are illustrative rather than prescribed.

The framework explicitly discourages treating numerical scoring as a substitute for judgment.

4.24 Strengths of AMTEF

The framework has several intended strengths.

First, it is multidimensional.

Second, it is adaptable.

Third, it explicitly includes uncertainty.

Fourth, it incorporates organizational readiness.

Fifth, it integrates technical and social evaluation.

Sixth, it includes lifecycle thinking.

Seventh, it recognizes governance as part of technology readiness.

Eighth, it supports continuous reassessment.

4.25 Limitations of AMTEF

No evaluation framework can eliminate all uncertainty.

Potential limitations include:

difficulty obtaining reliable evidence;

subjective stakeholder judgments;

difficulty assigning weights;

rapidly changing technologies;

cultural differences;

regulatory differences;

measurement limitations;

risk of oversimplifying complex social effects.

The framework should therefore be treated as a structured decision-support system rather than an objective machine that produces unquestionable answers.

4.26 Conditions for Effective Use

AMTEF should be implemented according to five principles.

Principle 1: Evidence

Claims should be supported by appropriate evidence.

Principle 2: Transparency

Assumptions and uncertainties should be documented.

Principle 3: Participation

Relevant stakeholders should have opportunities to contribute.

Principle 4: Proportionality

The depth of evaluation should correspond to the potential consequences of the technology.

Principle 5: Adaptability

Evaluation should be updated as evidence changes.

These principles align with contemporary approaches to technology assessment, which increasingly emphasize timely strategic intelligence, societal consequences and continuous anticipation.

4.27 Integrated Part I Model

The four chapters can now be integrated into one conceptual model.

Figure 4.2 — Foundations of Technology Evaluation

▼

▼

▼

▼

▼

▼

▼

▼

Conceptual pathway: The framework moves from understanding the technology, through technology assessment and recognition of evaluation difficulty, to multidimensional AMTEF evaluation, evidence-based decision-making, and an iterative cycle of implementation and reassessment.

4.28 Integrated Technology Evaluation Process

The complete Part I model can be summarized as:

Step 1: Understand

Identify the technology and its ecosystem.

Step 2: Assess maturity

Determine technological readiness and development stage.

Step 3: Understand consequences

Identify possible economic, social, human and environmental effects.

Step 4: Identify uncertainty

Document knowns, unknowns and assumptions.

Step 5: Evaluate dimensions

Apply the ten AMTEF dimensions.

Step 6: Compare alternatives

Compare with existing practices and alternative technologies.

Step 7: Determine conditions

Identify safeguards, governance requirements and organizational prerequisites.

Step 8: Pilot where appropriate

Generate additional evidence under controlled conditions.

Step 9: Implement

Deploy with monitoring and accountability.

Step 10: Reassess

Update the evaluation as technology and context change.

4.29 Chapter Research Questions

What dimensions should constitute a comprehensive technology-evaluation framework?

How can technical performance be distinguished from functional effectiveness?

How should economic, social and strategic value be incorporated into technology evaluation?

How can organizations evaluate technology when evidence is incomplete?

What role should stakeholder participation play in technology evaluation?

How can technological risks be incorporated without creating false numerical precision?

How can organizations determine whether they are ready to adopt a technology?

How can technology evaluation become a continuous organizational process?

4.30 Chapter Summary

This chapter introduced the Adams Multidimensional Technology Evaluation Framework.

AMTEF proposes that technology evaluation should incorporate ten dimensions:

technical performance;

functional effectiveness;

economic value;

organizational readiness;

human impact;

social consequences;

ethical responsibility;

environmental sustainability;

risk and security; and

governance and regulatory compatibility.

The framework is designed to help decision-makers move beyond technology-centred evaluation toward system-centred evaluation.

Its central proposition is that a technology should not be considered ready simply because it works.

Technology readiness requires consideration of whether the technology is technically viable, functionally effective, economically meaningful, organizationally achievable, human-centred, socially responsible, ethically defensible, environmentally sustainable, adequately secure and governable.

The framework also treats evaluation as continuous.

The decision to adopt a technology is therefore not the end of evaluation.

It is the beginning of a new stage of evidence generation.

4.31 Discussion Questions

Why should technology evaluation include ten dimensions rather than technical performance alone?

Should all dimensions receive equal weight?

What types of technology require mandatory threshold criteria?

How can organizations avoid false precision when assigning numerical technology scores?

How should stakeholder perspectives influence evaluation?

What are the limitations of AMTEF?

How could AMTEF be adapted to artificial intelligence?

How could AMTEF be adapted to renewable-energy technologies?

How could AMTEF be applied to biotechnology?

How could AMTEF be applied to higher education?