Part V — Technology Evaluation in Developing Economies, Responsible Technology Adoption, and the Future of Technology Evaluation

Chapter 18. Building a Responsible Technology Evaluation System

18.1 Introduction

The evaluation of technology should ultimately support responsible decision-making. Responsibility requires more than technical compliance. It involves evidence, transparency, accountability, stakeholder participation, risk management, ethical reflection, sustainability, and continuous monitoring.

Responsible innovation has been described as an approach that seeks to anticipate impacts, reflect on purposes and values, engage relevant stakeholders, and respond to emerging knowledge (Stilgoe et al., 2013). Technology evaluation provides an important operational mechanism for achieving these goals.

A responsible technology evaluation system should therefore answer five fundamental questions:

What are we developing or adopting?

Why are we developing or adopting it?

What evidence supports its expected benefits?

What risks and consequences may arise?

How will we monitor and govern it after implementation?

18.2 From Technology Assessment to Responsible Adoption

Traditional technology assessment has often concentrated on identifying the consequences, impacts, benefits and risks associated with a technology. Such assessment remains important, but contemporary technological environments require a broader perspective. Technologies increasingly operate within complex social, organizational, economic, environmental and regulatory systems. Consequently, the central question is no longer simply what consequences a technology might produce, but also whether, under what conditions, by whom and at what scale the technology should be adopted.

Responsible technology evaluation therefore creates a bridge between assessment and decision-making. Evaluation provides evidence about technical performance, economic feasibility, human consequences, environmental effects, organizational readiness, ethical considerations, security, governance and risk. Adoption decisions then use this evidence to determine whether a technology should be explored further, tested through a pilot, implemented under specified conditions, modified, delayed or rejected.

A responsible process can be represented as:

Technology Identification → Problem Definition → Evidence Collection → Multidimensional Evaluation → Stakeholder Deliberation → Risk Assessment → Pilot → Decision → Implementation → Monitoring → Re-Evaluation

The sequence should not be interpreted as completely linear. Evidence generated during a pilot may reveal that the original problem was incorrectly defined. Stakeholder consultation may identify an unrecognized risk. Monitoring may reveal that actual outcomes differ from predicted outcomes. New evidence may also change the original evaluation. Responsible adoption is therefore better understood as an iterative process in which information can move the organization backward and forward between stages.

Problem definition is particularly important because organizations sometimes begin with a technology and then search for a problem that appears to justify its acquisition. Responsible adoption reverses this tendency. The organization first establishes the problem, determines the desired outcomes and identifies possible approaches before selecting a technological intervention.

Evidence collection should also be multidimensional. Technical performance alone cannot establish whether a technology is appropriate. Financial, educational, social, environmental, ethical, organizational and governance evidence may all be necessary. The importance of each dimension will depend upon the characteristics and consequences of the technology.

Stakeholder deliberation provides a mechanism through which evidence can be interpreted within its social and institutional context. Different stakeholders may have different experiences and interests. A technology may increase organizational efficiency while simultaneously increasing workload for employees. It may improve service availability while creating accessibility barriers. Deliberation allows these dimensions to be examined rather than hidden behind a single performance measure.

Risk assessment should occur before implementation and continue throughout the lifecycle. A technology that appears low risk during initial testing may become higher risk when deployed at scale, connected to other systems or used in new contexts. Pilot implementation can therefore provide an intermediate stage between theoretical evaluation and full deployment.

Implementation should include clearly defined responsibilities, resources, governance arrangements, training and monitoring mechanisms. After implementation, evidence should be collected continuously. Monitoring should determine whether the technology is achieving its intended objectives and whether unexpected consequences are emerging.

This approach transforms technology evaluation into a lifecycle process. The final decision is therefore not the end of evaluation. Instead, implementation creates a new evidence environment in which assumptions can be tested against real-world experience.

18.3 The Responsible Technology Evaluation Cycle

The responsible technology evaluation cycle can be represented as follows:

Figure 18.1 — Responsible Technology Evaluation Cycle

Identify → Understand → Evaluate → Deliberate → Pilot → Decide → Implement → Monitor → Learn → Re-evaluate

The cycle is intentionally continuous because technology itself is continuously changing. Hardware, software, algorithms, data, regulations, user behaviour, cybersecurity threats, markets and environmental conditions may change after a technology has been introduced. An evaluation that was appropriate at the time of procurement may therefore become incomplete or outdated later.

The identification stage establishes the technology or technological opportunity under consideration. The organization should determine what the technology is intended to accomplish and why it has become relevant. At this stage, it is important to avoid assuming that technological availability creates a requirement for adoption.

The understanding stage examines the technology, its operating environment, dependencies, capabilities and limitations. Stakeholders should develop a sufficiently clear understanding of how the technology functions and how it interacts with existing systems and processes.

The evaluation stage applies appropriate evidence and analytical methods. Depending on the technology, this may involve technical testing, financial analysis, user research, environmental assessment, cybersecurity testing, privacy review, ethical assessment, organizational-readiness analysis or other methods.

The deliberation stage recognizes that evidence does not automatically determine a decision. Stakeholders must consider trade-offs, uncertainties, alternatives and competing objectives. Deliberation provides a structured environment for interpreting evidence and identifying conditions under which adoption may be responsible.

Pilot implementation can provide evidence under controlled or limited conditions. The purpose of a pilot is not simply to demonstrate that the technology functions, but to examine whether it produces the intended outcomes in the actual organizational or social environment.

The decision stage determines the appropriate course of action based on the available evidence. Possible decisions may include proceeding with implementation, modifying the technology, conducting additional evaluation, restricting deployment, delaying adoption or determining that another solution is more appropriate.

Implementation should occur with appropriate governance, training, monitoring and risk controls. Monitoring then produces new evidence about actual performance, user behaviour, costs, risks and outcomes.

Learning is essential because implementation can reveal information that was unavailable during the original assessment. Organizations should therefore use monitoring results to revise assumptions, improve procedures and modify the technology where necessary.

Re-evaluation completes the cycle by determining whether the technology remains appropriate. Re-evaluation may be triggered by changing regulations, new vulnerabilities, technological upgrades, new evidence, changing organizational objectives, changes in user populations, environmental conditions or significant incidents.

A technology should therefore not be regarded as permanently "approved" merely because it passed an initial evaluation. Approval should generally be understood as conditional upon the assumptions and circumstances under which the evaluation was conducted.

18.4 Principle 1: Purpose Before Technology

Responsible technology evaluation should begin with the problem rather than with the technology. Organizations frequently encounter technological products that are presented as innovative, transformative or essential. Commercial promotion, competitive pressure, institutional enthusiasm and technological fashion can create incentives to adopt systems before the underlying problem has been adequately defined.

Purpose-before-technology thinking reverses this sequence. The organization first asks what problem it is attempting to solve, what outcomes are required, who is affected by the problem, what causes or contributes to it, and whether technology is actually necessary.

The question of whether technology is required is particularly important. Some organizational problems may be better addressed through changes in policy, staffing, training, communication, workflow or management. Introducing technology into a poorly designed process can simply automate an inefficient process rather than solve the underlying problem.

Organizations should also identify the desired outcomes. A university introducing an AI-based student-support system, for example, should establish whether the intended objective is improved retention, faster intervention, better advising or more efficient use of support resources. Without a clearly defined purpose, it becomes difficult to determine whether the technology has succeeded.

Alternative solutions should be examined before adoption. These alternatives may include different technologies, organizational changes, human interventions or combinations of approaches. Comparing alternatives helps prevent technology-led decision-making.

The principle of purpose before technology therefore encourages organizations to distinguish between technological capability and genuine organizational need. A technology can be impressive without being appropriate. Responsible adoption requires evidence that the technology addresses a meaningful problem and contributes to clearly defined outcomes.

18.5 Principle 2: Evidence Before Scale

When uncertainty is substantial, technology should generally be tested before large-scale implementation. The purpose of testing is to generate evidence concerning actual performance, usability, risks, costs and consequences. Evidence can be generated through pilot projects, controlled studies, operational data, user testing, benchmarking, independent evaluations, case studies, simulations, expert review and longitudinal monitoring.

A pilot provides an opportunity to test assumptions under realistic conditions. It can reveal whether users understand the technology, whether existing systems can support it, whether expected benefits materialize and whether unexpected problems emerge. A pilot can also identify training requirements and organizational changes that were not anticipated during procurement.

Controlled studies can provide stronger evidence in circumstances where outcomes can be measured systematically. For some technologies, comparison with existing processes or alternative interventions may be appropriate. However, not every technology can or should be evaluated through a conventional experiment. Method selection should correspond to the evaluation question.

Operational data provide another valuable source of evidence. Real-world system performance may differ from vendor specifications or laboratory results. Actual uptime, maintenance requirements, user behaviour, error rates, support requests, costs and outcomes can provide evidence concerning the technology under genuine operating conditions.

User testing is particularly important when usability and accessibility are significant. A system may meet technical specifications while being difficult for intended users to navigate. Independent evaluation can provide additional protection against overreliance on vendor claims.

The amount and strength of evidence required should increase with potential risk. A low-risk administrative application may require relatively limited testing, whereas a technology affecting health, financial security, employment, public safety, educational progression or fundamental rights requires substantially stronger evidence.

Evidence before scale therefore represents a precautionary approach to uncertainty. It does not mean that every technology must undergo lengthy testing before any use. Rather, it means that the level of evidence should be appropriate to the consequences of being wrong.

18.6 Principle 3: Proportionality

Technology evaluation should be proportional to risk, uncertainty and scale. Not every technology requires the same degree of scrutiny. Applying identical evaluation requirements to every technology could consume resources unnecessarily for low-risk applications while potentially failing to distinguish technologies requiring much stronger oversight.

A useful conceptual relationship can be expressed as:

Required Evaluation Intensity ∝ Potential Harm × Uncertainty × Scale

This should not be interpreted as a mathematical law or a precise scoring formula. It is a decision principle that emphasizes the relationship between potential consequences and the level of evaluation required.

Potential harm refers to the seriousness of adverse consequences if the technology performs poorly or is misused. A system that controls access to routine information may have relatively limited consequences when it fails, whereas a system involved in healthcare decisions, financial transactions or public safety may have much more significant consequences.

Uncertainty concerns how much is known about the technology, its performance and its consequences. Emerging technologies may have limited historical evidence. Where uncertainty is high, additional testing, monitoring and staged implementation may be appropriate.

Scale concerns the number of people, organizations or systems affected. A small pilot may produce limited consequences if something goes wrong, whereas nationwide deployment can magnify even relatively small error rates.

Proportionality should also consider reversibility. If a decision can easily be reversed, evaluation requirements may differ from those associated with an irreversible or highly disruptive intervention. Technologies that create long-term environmental, legal, financial or social consequences may require particularly careful evaluation.

Proportionality therefore allows organizations to concentrate evaluation resources where they matter most. It supports rigorous governance without assuming that every technology presents the same level of risk.

18.7 Principle 4: Stakeholder Participation

Technology decisions affect multiple stakeholder groups, and responsible evaluation should therefore include meaningful stakeholder participation. Users, employees, managers, regulators, researchers, communities, suppliers, technical specialists, customers and other affected groups may experience the same technology in very different ways.

Technical experts may understand system architecture and performance, but they may not fully understand how a technology changes everyday work. Employees may identify workload implications that are invisible in technical specifications. Students may identify accessibility or usability problems. Communities may identify environmental or social consequences. Researchers may identify implications for data governance or scholarly integrity.

Participation is therefore an evidence-generating activity as well as a governance activity. Stakeholders can contribute information concerning actual practices, expectations, concerns and unintended consequences.

Participation should also be appropriately designed. Consultation should not simply involve informing stakeholders after a decision has already been made. Where appropriate, stakeholders should have opportunities to influence requirements, evaluation criteria, pilot design and implementation conditions.

Particular attention should be given to groups that may be disproportionately affected by a technology. Vulnerable populations may have less institutional power to influence decisions yet may experience greater consequences from technological errors or exclusion. Inclusive evaluation can therefore reveal risks that would otherwise remain invisible.

Stilgoe, Owen and Macnaghten (2013) describe responsible innovation in terms that include anticipation, reflection, inclusion and responsiveness. These concepts are relevant to technology evaluation because responsible decision-making requires organizations to consider possible consequences, examine assumptions, include relevant perspectives and respond to emerging evidence.

Stakeholder participation does not mean that every stakeholder preference must determine the final decision. Rather, it ensures that relevant experiences and knowledge are considered alongside technical and analytical evidence.

18.8 Principle 5: Transparency

Transparency means providing understandable information about a technology and the processes through which it is used. Relevant information may concern what the technology does, what data it uses, its limitations, its risks, who controls it, how decisions are made, how users can challenge outcomes and how incidents are handled.

Transparency is particularly important where users are affected by automated or technology-mediated decisions. Individuals may reasonably need to know that a system is being used and understand the general nature of its role in a decision.

Transparency does not necessarily require disclosure of proprietary source code or commercially confidential information. An organization can provide meaningful information about system purpose, inputs, outputs, decision processes, limitations, governance and accountability without revealing every technical detail.

The appropriate level of transparency should be related to the consequences of the technology. A simple administrative tool may require relatively straightforward documentation, while an AI system influencing employment, education, healthcare or financial decisions may require much more extensive information.

Transparency should also extend to limitations. Organizations should communicate not only what a technology can do but also what it cannot reliably do. Communicating uncertainty is an important component of responsible technology governance.

Incident transparency is also relevant. When significant errors or security events occur, organizations should have appropriate procedures for documenting, communicating and learning from them. Transparency therefore contributes to accountability and institutional learning.

18.9 Principle 6: Explainability

Explainability becomes particularly important when technology influences significant decisions affecting individuals or organizations. Examples include credit decisions, employment screening, student assessment, healthcare recommendations, insurance decisions and eligibility for public services.

The meaning of explainability can vary according to context. In some situations, users may require a simple explanation of the factors that influenced an outcome. In other situations, technical experts may require detailed documentation concerning data, model design, assumptions, validation and performance.

The required degree of explanation should therefore be proportional to the consequences of the decision. A recommendation concerning a low-risk administrative matter may not require the same level of explanation as an automated decision that affects a person's educational progression or access to an essential service.

Explainability also supports error correction. If an organization cannot understand why a system produced a particular result, it may be difficult to determine whether the result is appropriate or erroneous. Explainability can therefore contribute to accountability, auditing and improvement.

However, explainability should not be treated as an absolute requirement that every technical process must be completely understandable to every user. Some technologies are highly complex. The practical objective is to provide sufficient information to enable appropriate understanding, challenge, oversight and accountability.

18.10 Principle 7: Human Oversight

Human oversight should be proportionate to the risks associated with a technology. High-risk systems may require human approval, human review, override mechanisms, escalation procedures, independent auditing, continuous monitoring and formal incident reporting.

The presence of a human somewhere within a technological process does not automatically constitute meaningful human oversight. If a human operator is required to approve thousands of automated decisions without sufficient time, information or authority to question the system, the oversight may be largely symbolic.

Meaningful oversight requires capability and authority. Individuals responsible for reviewing technology-assisted decisions should understand the system sufficiently to recognize potential errors and should have the authority to challenge, modify or stop the process where appropriate.

Human oversight is particularly important where automated systems affect rights, opportunities, safety, education, employment, healthcare or financial circumstances. In such settings, technology should generally support rather than eliminate responsible human judgement.

Oversight should also extend beyond individual decisions to system-level monitoring. Organizations should examine whether error rates are changing, whether performance differs across groups, whether new vulnerabilities have emerged and whether the system remains appropriate for its original purpose.

UNESCO's Recommendation on the Ethics of Artificial Intelligence emphasizes principles including human oversight, human rights, dignity, fairness, transparency and accountability (UNESCO, 2021). These principles reinforce the idea that technological capability should remain embedded within appropriate human and institutional governance.

18.11 Principle 8: Accountability

Accountability requires clear identification of responsibility throughout the technology lifecycle. Organizations should establish who is responsible for procurement, implementation, data management, security, monitoring, incident response, compliance, user support and eventual decommissioning.

Ambiguous responsibility can create significant risks. When a technology produces an unexpected result, an organization should be able to determine who is responsible for investigating the event, communicating with affected parties, correcting the problem and preventing recurrence.

Accountability should extend across organizational and vendor relationships. Outsourcing a technological function does not necessarily eliminate the institution's responsibility for the consequences of using that technology. Organizations should therefore understand which responsibilities remain internal and which are assigned contractually to external providers.

Documentation is an important component of accountability. Institutions should maintain appropriate records concerning technology selection, evaluation results, approvals, configurations, policies, incidents and changes. These records create an institutional memory that supports auditing and future decision-making.

Accountability also requires escalation mechanisms. Users and employees should know where to report problems and how concerns will be investigated. In high-impact environments, independent review may also be appropriate.

A responsible technology system should therefore never create a situation in which every participant assumes that someone else is responsible. Accountability should be explicit, documented and supported by appropriate authority.

18.12 Principle 9: Privacy by Design

Privacy should be incorporated into technology development, procurement and implementation from the beginning rather than treated as a final compliance check. Privacy-by-design thinking requires organizations to consider how personal information will be collected, processed, stored, shared and eventually deleted before the system becomes operational.

Data minimization is an important principle. Organizations should collect only the information necessary for legitimate purposes. Collecting large quantities of data simply because they may become useful in the future can increase privacy and security risks.

Purpose limitation requires organizations to establish why information is being collected and to prevent inappropriate secondary uses. Information collected for one purpose should not automatically be repurposed for another substantially different purpose without appropriate consideration of legal, ethical and governance requirements.

Access control is another important component. Not every employee or system should have access to every category of information. Access should be based on legitimate requirements and should be monitored appropriately.

Retention also requires careful consideration. Organizations should determine how long information needs to be retained and establish appropriate procedures for secure deletion or anonymization where applicable.

Encryption and other security controls can reduce the risk of unauthorized access. However, privacy is broader than technical security. It also involves governance, transparency, user rights, organizational practices and responsible data use.

Consent may be relevant in some contexts, but it should not be treated as the only foundation of responsible data governance. Organizations should also consider purpose, necessity, proportionality, legal requirements, legitimate institutional interests and the reasonable expectations of affected individuals.

Cross-border data transfers require additional consideration where information is processed or stored across jurisdictions. Organizations should understand the applicable requirements and contractual arrangements.

Privacy evaluation should continue throughout the technology lifecycle. New data uses, software updates, integrations and changes in organizational purpose can alter the privacy implications of a system.

18.13 Technology Evaluation Governance Committee

Large organizations may establish a Technology Evaluation and Governance Committee.

Possible membership includes:

technology specialists;

business managers;

legal experts;

cybersecurity specialists;

ethics specialists;

environmental specialists;

representatives of users;

researchers;

quality-assurance personnel; and

senior leadership.

The committee should not simply approve technologies. It should establish evaluation standards and monitor performance.

18.14 Technology Evaluation Policy

An organizational technology evaluation policy can establish:

which technologies require evaluation;

who conducts evaluations;

minimum evidence requirements;

risk classifications;

stakeholder requirements;

pilot requirements;

approval authorities;

monitoring requirements;

review intervals;

incident-reporting procedures;

documentation requirements; and

decommissioning procedures.

18.15 Technology Risk Classification

Organizations can classify technologies according to potential impact.

Table 18.1 — Illustrative Technology Risk Classification

The categories should be adapted to the organization's context rather than treated as universal standards.

18.16 Evaluation Evidence Register

Every major technology evaluation should maintain an evidence register.

Table 18.2 — Technology Evidence Register

The evidence register helps prevent unsupported claims.

18.20 Uncertainty Register

Technology evaluation should explicitly document uncertainty.

Examples include:

uncertain demand;

uncertain costs;

uncertain performance;

uncertain environmental impacts;

uncertain regulation;

uncertain user behaviour;

uncertain cybersecurity threats;

uncertain technological development.

Uncertainty should not be hidden simply because decision-makers prefer certainty.

18.21 Decision Record

A responsible technology decision should produce a written decision record.

The record should document:

technology evaluated;

problem addressed;

alternatives considered;

evidence reviewed;

assumptions;

risks;

stakeholder views;

mitigation measures;

unresolved uncertainties;

decision;

responsible officers; and

review date.

This creates institutional memory.

18.22 The AMTEF Decision Architecture

The AMTEF decision architecture can be represented through ten dimensions:

Table 18.3 — AMTEF Responsible Evaluation Architecture

The framework is multidimensional rather than additive by default.

This distinction is important.

A serious ethical or safety problem should not necessarily be “cancelled out” by strong financial performance. Likewise, high economic value should not automatically compensate for unacceptable security risk.

18.23 The Principle of Non-Compensatory Risk

AMTEF therefore incorporates a non-compensatory principle for critical risks.

Where a technology presents a potentially unacceptable level of risk in areas such as safety, legality, human rights, privacy, or cybersecurity, strong performance in another dimension should not automatically justify deployment.

This does not mean that every risk must be eliminated. Risk is often unavoidable.

It means that critical risks require explicit treatment, mitigation, governance, and decision authority.

18.24 Monitoring After Implementation

Evaluation must continue after adoption.

Monitoring indicators may include:

technical performance;

user satisfaction;

cost;

productivity;

incidents;

security events;

complaints;

environmental impact;

equity;

unintended consequences;

regulatory changes; and

changes in technology capability.

This converts technology evaluation from an event into a learning system.

18.25 Re-Evaluation Triggers

A technology should be reconsidered when:

major software changes occur;

new functionality is introduced;

a serious incident occurs;

regulation changes;

new vulnerabilities are discovered;

user populations change;

significant costs increase;

performance declines;

environmental evidence changes; or

the technology is deployed in a new context.

18.26 Technology Sunset and Decommissioning

Responsible evaluation must also address the end of the technology lifecycle.

Questions include:

When should the technology be retired?

How will data be preserved or deleted?

How will users’ transition?

How will contracts be terminated?

What happens to hardware?

How will electronic waste be handled?

What happens if the vendor ceases operation?

A technology should have an exit strategy.

18.27 Responsible Technology Evaluation Scorecard

Table 18.4 — Responsible Technology Evaluation Scorecard

18.28 Institutionalizing AMTEF

AMTEF becomes most useful when incorporated into organizational processes.

It can be integrated into:

strategic planning;

procurement;

research ethics;

project management;

risk management;

quality assurance;

sustainability management;

cybersecurity;

academic governance;

investment decisions; and

organizational change management.

The goal is not to create another isolated administrative process.

The goal is to make multidimensional technology evaluation part of institutional decision-making.

18.29 Case Study: University Technology Governance

A university plans to introduce a generative AI platform for teaching and doctoral research.

The initial proposal emphasizes productivity.

AMTEF expands the evaluation.

Technical questions concern accuracy, reliability, integration, and system availability.

Economic questions concern licensing, training, support, and total cost.

Organizational questions concern faculty capability and institutional policies.

Human questions concern student learning, academic staff roles, and human oversight.

Social questions concern unequal access.

Ethical questions concern academic integrity, privacy, attribution, and fairness.

Environmental questions concern computing and infrastructure.

Risk questions concern data security and inappropriate outputs.

Governance questions concern responsibility, acceptable use, monitoring, and appeals.

The result is not merely a procurement decision. It is an institutional technology governance decision.

18.30 Research Questions for Chapter 18

How can organizations institutionalize technology evaluation?

What governance structures are appropriate for emerging technologies?

How should technology evaluation be proportionate to risk?

How can organizations integrate stakeholder participation?

How can uncertainty be documented and communicated?

How should technologies be monitored after adoption?

What conditions should trigger technology re-evaluation?

How can AMTEF be integrated into organizational governance?

18.31 Chapter Summary

Responsible technology evaluation represents a comprehensive approach to understanding, assessing, adopting, implementing, monitoring, and governing technology throughout its lifecycle. It recognizes that technology is not simply a technical object that can be evaluated according to functionality, efficiency, speed, accuracy, or cost. Technologies operate within social, organizational, economic, environmental, ethical, legal, and institutional contexts, and their consequences depend substantially on how they are designed, introduced, governed, used, maintained, modified, and eventually retired. Responsible technology evaluation therefore requires decision makers to examine not only what a technology can do, but also why it is being introduced, who will use it, who may be affected by it, what evidence supports its expected benefits, what risks may arise, and whether the organization has the capacity to manage those risks responsibly.

A central starting point is purpose. Responsible technology adoption should begin with a clearly defined problem, need, opportunity, or institutional objective rather than with enthusiasm for a particular technology. Organizations should first determine what they are attempting to achieve and then consider whether technology is necessary to achieve that objective. In some situations, a technological intervention may provide significant benefits, while in other situations organizational redesign, staff development, improved communication, revised procedures, additional resources, or other non-technological alternatives may provide more appropriate solutions. This purpose-first approach reduces the risk of adopting technology simply because it is new, fashionable, commercially promoted, or associated with innovation. It also encourages decision makers to evaluate whether the technology actually contributes to meaningful outcomes rather than merely increasing technological activity.

Evidence is another central foundation of responsible evaluation. Decisions about technology should be supported by evidence that is relevant to the specific context in which the technology will be used. Vendor claims, demonstrations, promotional materials, technical specifications, and laboratory results may provide useful information, but they should not automatically be treated as sufficient evidence for organizational adoption. Evidence may need to include pilot studies, user testing, independent evaluations, operational performance data, benchmarking, case studies, security assessments, financial analysis, environmental assessment, stakeholder feedback, and longitudinal monitoring. The appropriate level of evidence should reflect the potential consequences of the technology. Technologies associated with relatively minor and reversible consequences may require less extensive evaluation, whereas technologies that influence healthcare, education, employment, financial decisions, public services, safety, personal data, or other consequential areas require more extensive and rigorous evidence.

The principle of evidence before scale is particularly important. A technology may appear successful when tested with a small group or under controlled conditions but produce different results when implemented across a larger and more diverse population. Scaling can introduce new technical pressures, unexpected user behaviours, interoperability problems, cybersecurity vulnerabilities, resource constraints, organizational resistance, and financial costs. Pilot implementation therefore provides an opportunity to identify weaknesses and unintended consequences before substantial resources are committed. However, a pilot should not be regarded as automatically proving long-term effectiveness. The transition from pilot to large-scale implementation should be based on a reasoned assessment of the evidence, remaining uncertainties, risk exposure, organizational capacity, and conditions under which the technology is expected to operate.

Risk must be considered throughout technology evaluation rather than treated as a separate exercise conducted immediately before implementation. Risks may be technical, financial, operational, cybersecurity-related, environmental, legal, ethical, social, organizational, or reputational. Some risks may be predictable, while others may emerge only after the technology interacts with real-world users and existing systems. Responsible evaluation therefore considers both known risks and uncertainty. Where potential harm is significant and evidence is limited, stronger evaluation, additional safeguards, staged implementation, independent review, or delayed deployment may be appropriate. Risk evaluation should also consider reversibility. A technology that can easily be discontinued or replaced presents a different governance challenge from one that creates long-term dependency, restructures institutional processes, or produces consequences that cannot easily be reversed.

People remain central to responsible technology evaluation. Users, employees, students, customers, managers, researchers, communities, technical specialists, regulators, suppliers, and other affected groups may experience technology differently. Meaningful stakeholder participation can therefore provide information that technical testing alone cannot reveal. Stakeholders may identify usability problems, accessibility barriers, workload increases, privacy concerns, cultural issues, unintended discrimination, training requirements, or operational difficulties. Participation is not simply a mechanism for obtaining approval for a predetermined decision. It can contribute to understanding the technology itself and can improve the quality of the evidence used in decision making. Particular attention should be given to individuals and groups who may experience disproportionate effects from a technological intervention.

Responsible evaluation also requires attention to human agency. Technology should support people in making informed decisions rather than unnecessarily removing meaningful human judgement from important processes. This principle is particularly significant in artificial intelligence and automated decision systems. Automation may improve efficiency and consistency in appropriate circumstances, but automated outputs can contain errors, biases, incomplete information, or contextual limitations. Human oversight must therefore be meaningful rather than merely symbolic. Where people retain responsibility for decisions, they should have sufficient knowledge, information, authority, time, and institutional support to question, override, or escalate technological recommendations when necessary. Human oversight is especially important when technological systems influence consequential decisions affecting individuals or communities.

Transparency and explainability form another important dimension of responsible technology evaluation. Individuals and organizations should have an appropriate understanding of what a technology does, what information it uses, what its limitations are, what risks have been identified, and how decisions or outputs can be questioned. Transparency does not necessarily require disclosure of every technical detail or source-code element. Instead, the level of transparency should be proportionate to the significance of the technology and the consequences of its use. A system used for routine administrative support may require a different level of explanation from a system used to influence healthcare treatment, student assessment, employment decisions, financial services, or access to public services. Meaningful transparency should enable appropriate oversight, accountability, error correction, and informed participation.

Accountability provides the institutional foundation through which responsible technology can be maintained. Organizations should clearly identify who is responsible for technology procurement, evaluation, implementation, data governance, cybersecurity, user support, monitoring, incident response, compliance, and eventual retirement. Responsibility should not become unclear simply because a technology is supplied by an external vendor or operated through a cloud-based platform. Organizations may outsource technological functions, but they generally retain important responsibilities concerning the consequences of how those technologies are used. Clear accountability structures help prevent situations in which problems arise but no individual, department, committee, or institution has clear authority to respond.

Privacy and security should be incorporated into technology design and evaluation from the beginning. Privacy should be considered in relation to data collection, data minimization, access, retention, sharing, secondary use, cross-border transfer, deletion, and individual rights. Security should address issues such as authentication, authorization, encryption, vulnerability management, secure development, monitoring, backup, recovery, incident response, and supply-chain risks. Privacy and cybersecurity should not be regarded as technical issues that can be added after deployment. Weaknesses incorporated during system design can become expensive or difficult to correct later. Responsible evaluation therefore examines how privacy and security are maintained throughout the entire lifecycle of the technology.

Environmental and intergenerational responsibilities extend technology evaluation beyond immediate organizational benefits. Technologies require materials, energy, infrastructure, transportation, maintenance, replacement, and eventual disposal. Digital technologies, artificial intelligence systems, data centres, electronic devices, batteries, vehicles, industrial equipment, and communication infrastructure may all have environmental consequences across their lifecycles. Responsible evaluation therefore considers resource consumption, greenhouse-gas emissions, water requirements, critical minerals, waste generation, repairability, reuse, recycling, and end-of-life management. It also considers whether today's technological choices restrict future generations' environmental, economic, or technological options. This longer-term perspective connects responsible technology evaluation with sustainability and intergenerational responsibility.

Equity and inclusion are equally important. A technology that performs well technically may nevertheless produce unequal outcomes if access, affordability, digital literacy, language, disability, geography, or socioeconomic conditions are ignored. Responsible evaluation asks whether different groups can realistically access and use the technology and whether some groups may experience greater risks or disadvantages. Accessibility should therefore be incorporated into technology selection, procurement, design, implementation, and monitoring. The evaluation of technology should consider not only the average user but also those who may face barriers to participation or may be disproportionately affected by system failures or automated decisions.

Responsible adoption also requires organizational readiness. Even technically capable technology can fail when an institution lacks the skills, infrastructure, leadership, financial resources, policies, support systems, or organizational culture required for effective implementation. Evaluation should therefore examine whether employees have the necessary competencies, whether users can obtain adequate training and support, whether infrastructure can sustain the system, whether existing processes need to be redesigned, and whether leadership is prepared to manage organizational change. Adoption should not be interpreted simply as the installation or purchase of a technology. Effective adoption occurs when the technology becomes integrated into organizational practices in a way that produces the intended outcomes while maintaining appropriate safeguards and accountability.

Governance provides the mechanism through which these principles can be coordinated. Responsible technology governance should establish who evaluates technologies, what criteria are applied, what evidence is required, who has decision-making authority, how risks are documented, how stakeholders participate, and how implementation is monitored. Governance should be proportionate to the technology's risk, scale, complexity, uncertainty, and potential consequences. A small administrative application should not necessarily require the same governance structure as an artificial intelligence system used in healthcare, education, employment, finance, or public administration. Proportional governance allows organizations to maintain appropriate control without creating unnecessary administrative burdens.

An important characteristic of responsible technology governance is that approval should not necessarily be considered permanent. Technologies change after deployment. Software is updated, algorithms are modified, vendors change their practices, new vulnerabilities are discovered, regulations evolve, users change, organizational conditions shift, and new evidence becomes available. A technology that was appropriate at the time of initial evaluation may therefore require reassessment later. Continuous evaluation creates a mechanism through which organizations can learn from operational experience and adjust their decisions accordingly. Monitoring should examine not only whether the technology continues to function technically but also whether it continues to achieve its intended educational, organizational, financial, social, environmental, and ethical objectives.

Re-evaluation is particularly important when significant changes occur. These may include major software or algorithmic updates, changes in data sources, cybersecurity incidents, new regulatory requirements, significant changes in user populations, unexpected performance outcomes, major increases in scale, new evidence concerning risks, or changes in the organization's strategic objectives. Re-evaluation should also occur when a technology becomes embedded in processes that were not anticipated during the original assessment. Such developments may change the nature and magnitude of the technology's consequences.

Responsible technology evaluation therefore operates as a learning system. The objective is not merely to approve or reject a technology at a single point in time. Instead, evaluation should generate knowledge that improves future decisions. Monitoring results, user feedback, incidents, complaints, performance data, financial outcomes, environmental indicators, and independent assessments can all contribute to organizational learning. Lessons from one technology can also inform the evaluation of future technologies. In this sense, responsible evaluation becomes part of institutional learning and continuous improvement.

The principle of proportionality connects these elements by recognizing that evaluation intensity should correspond to potential harm, uncertainty, scale, and reversibility. A technology with low potential harm, high reversibility, and substantial evidence may require relatively straightforward evaluation. By contrast, a technology with high potential consequences, substantial uncertainty, large-scale deployment, sensitive data, or limited reversibility requires more comprehensive assessment and stronger governance. This approach allows organizations to direct evaluation resources toward areas where they are most needed.

Responsible technology evaluation also requires recognition that no single evaluation method can capture every relevant dimension. Quantitative analysis can provide information about performance, costs, reliability, usage, and outcomes, while qualitative research can reveal user experiences, institutional culture, ethical concerns, contextual factors, and unintended consequences. Risk assessment can identify vulnerabilities, lifecycle assessment can examine environmental consequences, cost analysis can examine financial sustainability, and stakeholder engagement can identify perspectives that may otherwise be overlooked. The AMTEF approach can therefore be understood as an integrative framework in which multiple forms of evidence contribute to a more comprehensive evaluation rather than as a requirement to reduce every dimension of technology to a single numerical score.

The ultimate purpose of responsible technology evaluation is not to prevent technological innovation. Rather, it is to ensure that innovation is connected to meaningful human and organizational purposes and is introduced with appropriate evidence, safeguards, accountability, and learning mechanisms. Responsible evaluation creates conditions in which beneficial technologies can be adopted while uncertainty, unintended consequences, inequities, environmental effects, and governance weaknesses are recognized and managed. It also encourages organizations to reconsider technologies when evidence demonstrates that expected benefits are not being achieved or when the risks have become unacceptable within the relevant context.

The central principle of this chapter can therefore be expressed as follows: Responsible adoption requires evidence before scale, governance during implementation, and re-evaluation throughout the technology lifecycle. Evidence before scale means that organizations should establish a reasonable basis for believing that a technology can achieve its intended purpose before committing to extensive deployment. Governance during implementation means that responsibility, oversight, transparency, privacy, security, equity, human agency, and accountability must continue after adoption rather than ending when procurement or installation is completed. Re-evaluation throughout the technology lifecycle means that organizations must remain willing to learn, adapt, modify, restrict, replace, or retire technologies when circumstances or evidence change.

Ultimately, responsible technology evaluation represents a shift from the question, “Can this technology be implemented?” to a broader set of questions concerning whether it should be implemented in a particular context, for a particular purpose, under particular conditions, with particular safeguards, and with appropriate mechanisms for monitoring and accountability. This broader perspective recognizes that technological capability alone does not determine technological value. A technology becomes responsible in practice when its capabilities are aligned with legitimate purposes, evidence, human needs, institutional capacity, ethical principles, environmental responsibilities, and effective governance. Through this lifecycle-oriented approach, technology evaluation becomes not merely a technical assessment exercise but an ongoing process of responsible decision making, organizational learning, and sustainable technological development.