Chapter 15. Methods and Tools for Technology Evaluation — Part 2 of 4
15.16 Observation
Observation can identify differences between formal procedures and actual practice. This is particularly important in technology evaluation because users frequently adapt systems to fit their working environments. Technology designers and organizational managers may describe a particular workflow, while actual users may follow a substantially different process.
Employees may develop workarounds, bypass certain features, create parallel processes, maintain unofficial records or use additional technologies in ways not anticipated by system designers. These behaviours may indicate usability problems, organizational constraints, inadequate training or limitations in the technology itself.
Observation can therefore reveal important implementation information that surveys and interviews may not capture. Participants may not always recognize or report routine workarounds because they have become normalized. Direct observation can make these practices visible.
Observation can be structured or relatively open. Structured observation uses predefined categories or behaviours, whereas less structured observation allows researchers to document emerging activities and contextual events. The choice depends on the evaluation question.
For example, an evaluator examining an electronic health-record system might observe how healthcare professionals enter patient information, retrieve records, move between different screens and respond to interruptions. The evaluator may discover that clinicians use paper notes temporarily because the digital workflow is too slow during patient consultations. Such a finding may not be captured adequately by a general satisfaction survey.
Observation should also consider the environment surrounding technology use. Physical workspace, connectivity, interruptions, staffing levels, organizational hierarchy and time pressure may influence how a technology is used. A system that functions effectively in a controlled demonstration may operate differently under real workplace conditions.
Ethical considerations are particularly important where observation involves employees, patients, students or other individuals. Participants should understand the purpose of observation and the manner in which data will be recorded, stored and used. Privacy and confidentiality should be protected.
Within AMTEF, observation provides evidence about the difference between designed technology and technology-in-use. This distinction is essential because the actual consequences of technology often emerge through interaction between technical systems and human practice.
15.17 Case Study Research
Case-study research allows technologies to be examined within real organizational and social contexts. Yin (2018) emphasizes the value of case studies for investigating contemporary phenomena within their real-world environments, particularly where the boundaries between the phenomenon and its context are not clearly separated.
Case studies are particularly useful when technology cannot easily be separated from its organizational environment. A technological implementation may depend upon leadership, culture, infrastructure, policies, employee skills, financial resources, governance arrangements and external regulatory conditions. Studying the technology independently of these factors may therefore produce an incomplete evaluation.
A university implementing an artificial intelligence platform, for example, can be studied as a case involving the technology, faculty, students, policies, infrastructure, leadership, institutional culture, academic-integrity arrangements and data-governance procedures. The case can investigate not only whether the platform functions technically but also how the university adapts to it.
Case studies may be single-case or multiple-case designs. A single-case study may provide detailed examination of an unusual, critical or particularly informative implementation. Multiple-case research can allow researchers to examine similarities and differences across organizations, departments, technologies or implementation contexts.
Case studies can use multiple sources of evidence, including interviews, documents, observations, system data, performance measures, financial records, policy documents and surveys. This makes them particularly compatible with triangulation. The researcher can compare information from different sources rather than relying on a single account.
Case-study analysis should nevertheless remain systematic. The presence of multiple sources does not automatically guarantee methodological quality. Researchers should establish clear case boundaries, define the units of analysis, document data sources, maintain a transparent chain of evidence and explain how conclusions were derived.
Case studies can also provide valuable contextual knowledge for technology adoption. A technology that succeeds in one organization may not produce identical results elsewhere because organizational structures, resources, skills and cultures differ. Case-study research can therefore help identify contextual conditions that influence technological outcomes.
Within AMTEF, case-study research is particularly valuable because the framework is concerned with technology as a multidimensional and contextual phenomenon. The case-study approach enables technical, organizational, economic, social, environmental, ethical and governance dimensions to be examined within the setting in which the technology actually operates.
15.18 Cost-Benefit Analysis
Cost-benefit analysis compares the monetary value of costs with the monetary value of benefits associated with a technology, intervention or investment. It is widely used to support investment and policy decisions because it provides a structured way of examining whether anticipated or observed benefits justify the resources required.
Technology-related costs may include acquisition, installation, implementation, infrastructure, integration, training, maintenance, cybersecurity, licensing, staffing, data management, governance, upgrades and eventual replacement. Evaluators should avoid focusing exclusively on the purchase price because the initial acquisition cost may represent only one component of total cost over the technology lifecycle.
Benefits may include increased revenue, reduced operating costs, productivity improvements, reduced errors, improved service delivery, avoided losses, improved resource utilization and other outcomes that can reasonably be assigned monetary values. In some cases, benefits may also include avoided environmental or social costs where credible valuation methods are available.
Time is an important consideration in cost-benefit analysis. A technology may require substantial investment at the beginning while generating benefits over many years. Discounting may therefore be used to compare costs and benefits occurring at different points in time. The choice of discount rate can influence results and should be transparently reported.
Sensitivity analysis is also important because many technology costs and benefits are uncertain. Energy prices, maintenance costs, adoption levels, technology lifetimes, productivity gains and replacement schedules may differ from initial assumptions. Evaluators should therefore examine how results change under alternative assumptions.
A major limitation is that not every consequence can easily be converted into monetary terms. Privacy, dignity, trust, environmental quality, social cohesion, cultural value and some forms of safety may resist straightforward monetization. Assigning monetary values to such outcomes can introduce substantial assumptions and may create an appearance of precision that exceeds the quality of the underlying evidence.
Cost-benefit analysis should therefore be integrated with other evaluation methods rather than used as the sole decision criterion. A technology may produce a positive monetary balance while creating significant ethical or environmental concerns. Conversely, a technology with limited direct financial benefits may produce important social or public-value outcomes.
Within AMTEF, cost-benefit analysis should therefore be treated as one component of economic evaluation within a broader multidimensional assessment. Financial attractiveness should be examined alongside technical performance, organizational feasibility, risk, sustainability, social consequences, ethics and governance.
15.19 Cost-Effectiveness Analysis
Cost-effectiveness analysis compares costs with measurable outcomes rather than attempting to convert all outcomes into monetary values. It is particularly useful in healthcare, education, public services and other contexts where outcomes can be measured using meaningful units but are difficult or inappropriate to express entirely in monetary terms.
For example, rather than asking only how much a healthcare technology costs, an evaluator may examine the cost associated with each additional successful treatment, additional accurately detected condition, avoided hospitalization or other clinically meaningful outcome. In education, an evaluator might examine the cost associated with achieving a defined improvement in learning outcomes. In energy management, the analysis might compare the cost required to achieve a defined reduction in energy consumption.
The effectiveness measure must be carefully selected. It should correspond directly to the purpose of the technology and should be sufficiently reliable to support comparison. Using an inappropriate outcome measure can make the analysis misleading even when the mathematical calculations are correct.
Cost-effectiveness analysis can also compare alternative technologies that pursue the same objective. For example, two healthcare technologies may have different acquisition and operating costs but produce different levels of clinical effectiveness. The evaluator can examine the additional cost associated with the additional effectiveness achieved by one alternative compared with another.
An important distinction exists between cost-effectiveness and cost-benefit analysis. Cost-benefit analysis attempts to express both costs and benefits in monetary terms, whereas cost-effectiveness analysis normally leaves the outcome in its substantive unit. Cost-effectiveness analysis can therefore avoid some of the difficult assumptions associated with monetizing outcomes such as health improvement or educational achievement.
Nevertheless, cost-effectiveness analysis also has limitations. A technology may appear cost-effective when evaluated against one outcome while producing undesirable effects in another dimension. A healthcare technology might reduce treatment costs but create privacy concerns. An educational platform might improve test performance while increasing student stress or reducing accessibility for some users.
Cost-effectiveness results may also depend on the perspective adopted. An analysis conducted from the perspective of an organization may produce different results from an analysis conducted from the perspective of patients, employees, government or society as a whole. The evaluator should therefore clearly identify whose costs and outcomes are being considered.
Uncertainty should also be incorporated. Estimates of effectiveness, implementation cost, maintenance requirements and technology lifetime may be uncertain. Sensitivity analysis can therefore be used to determine whether conclusions remain stable under plausible alternative assumptions.
Within AMTEF, cost-effectiveness analysis provides an important economic tool for comparing technologies where meaningful outcomes can be measured but comprehensive monetization is inappropriate. It should be interpreted alongside technical, organizational, social, environmental, ethical and governance evidence to ensure that economic efficiency does not become detached from the broader consequences of technology.
15.20 Total Cost of Ownership
Total Cost of Ownership, commonly referred to as TCO, examines the complete cost associated with acquiring, implementing, operating, maintaining, modifying and eventually replacing or retiring a technology. TCO is particularly important because the purchase price of a technology often represents only a portion of its actual financial burden. A technology that appears inexpensive at the point of acquisition may become considerably more expensive when implementation, training, integration, maintenance, cybersecurity, upgrades and replacement are taken into account.
The TCO perspective therefore extends the economic evaluation across the technology lifecycle. Initial costs may include procurement, installation, configuration, customization, infrastructure preparation and integration with existing systems. Continuing costs may include software licences, subscriptions, cloud services, technical support, staffing, maintenance, data storage, cybersecurity, insurance, energy consumption and system monitoring. Further costs may arise when systems require upgrades, migration, replacement of hardware or retraining of personnel.
Information technologies provide a particularly important example. A software platform may have a relatively low initial acquisition price but require annual subscriptions, cloud-storage charges, integration services, cybersecurity controls, user training and technical support. An organization that evaluates only the initial purchase price may therefore significantly underestimate the long-term financial commitment.
TCO should also consider indirect costs. These may arise from employee time spent learning the technology, temporary productivity losses during implementation, disruption to established workflows, system downtime and the need to operate legacy systems while a new technology is being introduced. In some situations, organizations must maintain two systems simultaneously during a transition period, creating additional financial and operational costs.
Downtime can be particularly significant. If a technology failure interrupts business operations, the resulting costs may include lost productivity, delayed services, lost revenue, contractual penalties, reputational damage and additional recovery expenses. For critical infrastructure, the consequences may extend beyond financial loss to safety, public service continuity or organizational resilience.
The useful life of the technology should also be considered. A system with a low initial cost but a short useful life may have a higher lifecycle cost than a more expensive system that remains operational for a longer period. Similarly, technologies that become obsolete rapidly may require more frequent replacement and migration.
TCO analysis should distinguish between fixed and variable costs where appropriate. It should also identify which costs are certain and which are estimated. This distinction becomes particularly important when evaluating emerging technologies for which long-term maintenance and replacement costs may be uncertain.
TCO can also be examined from different organizational perspectives. A technology may be inexpensive for the technology department but expensive for end users because of additional workload or training requirements. Conversely, an investment in infrastructure may increase the organization's capital expenditure while reducing operating costs over time.
Within AMTEF, TCO should therefore be integrated with technical, organizational, environmental and risk evaluation. The lowest TCO does not necessarily indicate the most appropriate technology because financial cost is only one dimension of technology value. A technology with a higher lifecycle cost may provide substantially greater reliability, security, accessibility, sustainability or strategic value. TCO should therefore inform rather than replace multidimensional evaluation.
15.21 Multi-Criteria Decision Analysis
Multi-Criteria Decision Analysis, commonly known as MCDA, provides a structured approach for comparing technological alternatives across multiple criteria. Technology decisions frequently involve several objectives that cannot easily be reduced to a single measure. Criteria may include acquisition cost, operating cost, reliability, safety, technical performance, environmental impact, social acceptance, cybersecurity, organizational feasibility, scalability, regulatory compatibility and long-term sustainability.
MCDA is particularly relevant to AMTEF because technology evaluation is inherently multidimensional. A decision concerning an energy technology, for example, may require simultaneous consideration of efficiency, cost, reliability, land requirements, emissions, resource availability, community acceptance and energy-security implications. No single indicator can adequately represent all these dimensions.
Important foundations for multi-criteria decision approaches can be found in the work of Keeney and Raiffa (1993) and Belton and Stewart (2002). MCDA approaches can take different forms, ranging from relatively simple weighted criteria models to more sophisticated decision-analysis techniques. The appropriate approach depends on the complexity of the decision, the quality of available evidence and the purpose of the evaluation.
The first stage generally involves identifying the alternatives and evaluation criteria. Criteria should be relevant to the decision and defined sufficiently clearly that evaluators can determine how evidence will be interpreted. Criteria may be quantitative, qualitative or a combination of both. For example, annual operating cost may be expressed numerically, whereas organizational feasibility may be assessed through a structured qualitative scale.
The assignment of weights is one of the most important methodological issues in MCDA. Weighting criteria involves assumptions concerning their relative importance. If cost is assigned a high weight, a relatively inexpensive technology may receive greater overall preference within the mathematical model. If reliability or environmental performance receives greater weight, another alternative may produce a different result. The evaluator should therefore make the weighting process transparent.
Stakeholder participation can be useful when determining criteria and weights, particularly when technological decisions affect multiple groups. However, stakeholder preferences may differ substantially. Senior managers may prioritize financial sustainability, technical staff may emphasize reliability and maintainability, users may prioritize usability, while communities may emphasize environmental and social consequences.
Sensitivity analysis is therefore essential. Evaluators should examine whether changing reasonable assumptions concerning weights, measurements or criteria produces substantial changes in the resulting comparison. If small changes produce major changes in the result, the evaluation should communicate this uncertainty rather than presenting the output as definitive.
MCDA should also distinguish between evidence and judgement. Measured system performance is evidence, whereas the decision concerning how important that performance should be is partly a value judgement. Both are legitimate components of decision-making, but they should not be confused.
MCDA should therefore be used to structure deliberation rather than create a false impression of mathematical certainty. A numerical score generated by an MCDA model does not transform uncertain evidence or subjective assumptions into objective truth. Within AMTEF, MCDA is best understood as a transparent framework for organizing multidimensional evidence and making assumptions visible.
15.22 Technology Readiness Assessment
Technology readiness assessment examines the maturity of a technological system and the extent to which it has progressed from an early concept toward reliable operational deployment. Technology Readiness Levels, or TRLs, are commonly associated with staged assessments of technological maturity, although different sectors may adapt readiness frameworks to their specific circumstances.
Technical maturity, however, is only one dimension of readiness. A technology can be technically mature while remaining unsuitable for immediate implementation because other conditions have not been satisfied. Technology readiness should therefore be considered alongside organizational readiness, market readiness, regulatory readiness, infrastructure readiness, financial readiness and workforce readiness.
For example, an artificial intelligence system may demonstrate strong technical performance but have low regulatory readiness because necessary approvals or compliance requirements have not been completed. A medical device may function effectively in controlled testing but remain unsuitable for widespread deployment because clinical training, certification or reimbursement arrangements are incomplete.
A technology may also have high technical readiness but low market readiness. Customers may not understand the technology, complementary infrastructure may be unavailable, purchasing processes may not support it, or users may not perceive sufficient value to justify adoption.
Organizational readiness is equally important. An organization may acquire a highly mature technological system but lack the skills, leadership, governance structures, data infrastructure or financial resources necessary to operate it effectively. The resulting implementation failure should not automatically be interpreted as evidence that the technology itself is technically inadequate.
Readiness can therefore be represented as a multidimensional profile. Instead of asking simply whether a technology is “ready,” evaluators can examine the readiness of the technology, organization, market, infrastructure, regulatory environment, workforce, financing arrangements and supporting ecosystem.
Readiness is also dynamic. A technology may move rapidly from experimental development to commercial maturity, while regulatory, workforce or infrastructure readiness may develop more slowly. Conversely, a technology may be technically mature but lose readiness because supporting infrastructure becomes obsolete or critical supply chains become constrained.
Within AMTEF, readiness assessment should therefore identify both technological maturity and the conditions necessary for responsible implementation. A high technical readiness level should not automatically be interpreted as evidence that widespread adoption is appropriate. Rather, it should be treated as one component of a broader readiness assessment.
15.23 Life-Cycle Assessment
Life-Cycle Assessment, commonly abbreviated as LCA, is an established method for evaluating environmental impacts across the lifecycle of products, services and technological systems. Rather than examining only the operational phase of a technology, LCA considers environmental consequences across relevant stages from raw-material extraction through manufacturing, transportation, use, maintenance and end-of-life management.
The standard LCA process generally includes goal and scope definition, inventory analysis, impact assessment and interpretation. ISO 14040 and ISO 14044 provide important international standards for lifecycle assessment (ISO, 2006a, 2006b). These standards emphasize systematic definition of the assessment, transparent methodology and appropriate interpretation of results.
Goal and scope definition establishes why the LCA is being conducted, what system is being evaluated, the functional unit used for comparison and the boundaries of the assessment. These decisions are important because different boundaries can produce different findings.
Inventory analysis involves collecting information about inputs and outputs throughout the defined system. Inputs may include raw materials, energy, water and other resources, while outputs may include emissions, waste and other environmental releases. Data quality can significantly influence the reliability of the final assessment.
Impact assessment translates inventory information into relevant environmental impact categories. Depending on the purpose of the study, these may include greenhouse-gas emissions, climate change impacts, resource depletion, water use, eutrophication, acidification and other environmental categories.
LCA can reveal consequences that are not visible when only the operational phase is considered. An electric technology, for example, may have low operational emissions while requiring substantial resources during manufacturing. Similarly, a highly efficient digital system may consume less operational energy but depend on hardware containing materials with significant extraction and processing impacts.
System boundaries are therefore critical. If manufacturing is excluded from one assessment but included in another, direct comparisons may be misleading. The same applies to assumptions concerning electricity sources, product lifetime, recycling rates, transportation distances and maintenance.
LCA also requires careful treatment of uncertainty. Environmental inventories may contain estimated rather than directly measured values. Different assumptions concerning technology lifetime, recycling, energy mix or material sourcing can affect results. Sensitivity analysis can therefore help identify which assumptions have the greatest influence.
Within AMTEF, LCA represents an important environmental evaluation method that can be integrated with economic, technical, social and governance analysis. It helps prevent organizations from evaluating technologies only according to their immediate operating performance while overlooking environmental consequences occurring elsewhere in the lifecycle.
15.24 Risk Assessment
Risk assessment involves systematic identification, analysis and evaluation of potential events or conditions that could produce undesirable consequences. Technology-related risks may concern safety, cybersecurity, privacy, financial loss, operational disruption, environmental damage, regulatory non-compliance, reputational consequences or harm to users and communities.
Risk assessment can involve both qualitative and quantitative approaches. Qualitative methods may classify risks according to categories such as low, moderate or high likelihood and consequence. Quantitative methods may estimate probabilities, probability distributions, expected losses or other numerical measures where adequate data exist.
The choice between qualitative and quantitative approaches should depend on the availability and quality of evidence. Quantifying a risk does not necessarily make the assessment more accurate if the probability estimates are based on weak assumptions. In some emerging technology contexts, scenario-based qualitative assessment may therefore be more defensible than highly precise numerical estimates.
Failure Mode and Effects Analysis, or FMEA, examines potential failure modes, their causes and their consequences. It is particularly useful for systematically considering how individual components or processes could fail. The method can help organizations identify areas requiring additional controls, monitoring or redesign.
Fault Tree Analysis takes a different perspective. It begins with an undesired top-level event and works backward to identify combinations of contributing failures. This can be useful for analyzing complex technological systems in which several failures may interact to produce a major event.
Bow-Tie Analysis provides a visual representation linking potential causes to a central undesired event and then connecting that event to possible consequences. Preventive barriers can be examined on one side, while mitigating or recovery barriers can be examined on the other. This makes the approach useful for understanding how risk controls are intended to prevent or limit consequences.
Scenario analysis can be particularly useful where probabilities are difficult to estimate but plausible future conditions can be described. Stress testing can similarly examine how a technological system performs under unusually difficult conditions. Examples include extreme demand, infrastructure failure, cyberattack, supply disruption, energy-price changes or environmental stress.
Risk assessment should also consider interactions among risks. A technology may have several individually moderate risks that become significant when combined. A cloud-service outage, for example, may become more consequential when combined with inadequate backup procedures, insufficient staff capability and weak business-continuity arrangements.
Risk assessment should therefore not be limited to identifying isolated technical failures. Within AMTEF, it should examine technological, organizational, environmental, social, financial, cybersecurity, ethical and governance risks and consider how these risks may interact throughout the technology lifecycle.
15.25 Delphi Method
The Delphi method uses structured rounds of consultation with experts to investigate issues where evidence may be incomplete, uncertain or emerging. Participants generally provide judgements independently, receive summarized or aggregated feedback concerning the group's responses and may revise their views during subsequent rounds.
Delphi can be useful when empirical evidence is limited, technologies are emerging or expert knowledge is particularly important. For example, experts may be consulted concerning future applications of artificial intelligence, anticipated workforce requirements, emerging cybersecurity threats, future energy technologies or potential technological standards.
A central strength of the Delphi approach is that it can reduce some forms of direct group pressure because participants provide responses independently rather than reaching a conclusion through face-to-face discussion. Repeated rounds allow participants to reconsider their views in light of the aggregated responses of other experts.
Expert selection is therefore critical. The evaluator should establish appropriate criteria for expertise and provide sufficient information concerning the purpose and scope of the consultation. Experts may be selected based on academic knowledge, professional experience, technical specialization, policy experience or practical involvement with the technology.
Question design is equally important. Ambiguous questions can produce inconsistent responses and make consensus difficult to interpret. Researchers should clearly distinguish between questions asking about current conditions, future expectations, technical feasibility, social acceptance or normative preferences.
Consensus should also be interpreted carefully. A high level of agreement among experts does not necessarily demonstrate that a proposition is empirically true. Experts can share the same assumptions and still be mistaken. Delphi findings therefore represent structured expert judgement rather than empirical proof.
The selection of experts may also influence the results. A panel dominated by one professional discipline may produce different conclusions from a multidisciplinary panel. AMTEF-oriented Delphi studies should therefore consider whether technical, organizational, economic, environmental, social, ethical and governance perspectives need to be represented.
The reporting of Delphi studies should describe the number and characteristics of participants, selection criteria, number of rounds, response rates, definitions of consensus and methods used to aggregate responses. This transparency allows readers to assess the credibility and limitations of the findings.
Within AMTEF, Delphi can be particularly useful as a foresight and evidence-development method, especially when evaluating emerging technologies for which historical performance data are unavailable. It should nevertheless be integrated with empirical evidence whenever such evidence becomes available.
15.26 Scenario Planning
Scenario planning explores plausible future conditions rather than attempting to predict a single future. It is particularly useful for technologies operating within environments characterized by uncertainty, rapid change and interacting external forces.
Technological performance may be influenced by policy changes, climate conditions, market developments, demographic trends, technological convergence, geopolitical developments, infrastructure availability and changing user expectations. Scenario planning enables evaluators to examine how a technology might perform under different combinations of these conditions.
Scenarios may represent optimistic, constrained, disruptive or transformational conditions. The purpose is not necessarily to identify which scenario will occur but to explore how different conditions could influence technological value, risk and feasibility.
For example, an organization considering a large digital infrastructure investment could examine scenarios involving rapid growth in demand, constrained energy supply, major cybersecurity incidents, regulatory changes or accelerated technological obsolescence. Each scenario could then be evaluated against technical, financial, organizational and resilience criteria.
Scenario planning can also reveal vulnerabilities. A technology that performs well under normal conditions may become unsuitable under extreme conditions. An energy system, for example, may operate effectively under average demand but become vulnerable during prolonged heatwaves or supply disruptions.
Scenario planning is particularly useful for long-lived infrastructure because investments may remain operational for decades. Decision-makers must therefore consider not only current conditions but also the possibility of significant changes during the technology's expected operating life.
Scenarios should nevertheless be developed systematically rather than simply representing optimistic and pessimistic guesses. They should be based on identifiable drivers, uncertainties and evidence. The assumptions underlying each scenario should be documented clearly.
Within AMTEF, scenario planning can be connected with risk assessment, technology roadmapping, environmental analysis and strategic planning. It allows evaluators to examine whether a technology remains viable under multiple plausible futures rather than evaluating it exclusively against present conditions.
15.27 Horizon Scanning
Horizon scanning involves the systematic identification and examination of emerging technologies, trends, weak signals, developments and potential disruptions. It can support organizations in identifying technological opportunities and risks before they become mainstream.
Horizon scanning is particularly relevant in rapidly changing technological environments. Developments in artificial intelligence, biotechnology, quantum technologies, robotics, cybersecurity, energy storage and digital infrastructure can alter organizational capabilities and competitive conditions within relatively short periods.
The purpose of horizon scanning is not simply to create a list of new technologies. It involves examining the potential significance of emerging developments, the evidence supporting them, the barriers to adoption and the possible implications for the organization or wider system.
Weak signals can sometimes indicate developments that are not yet visible in mainstream technology markets. These may include early research publications, patents, regulatory consultations, pilot projects, investment patterns, changes in professional practice or emerging user communities.
Horizon scanning should nevertheless distinguish between speculative concepts and technologies supported by credible evidence. Not every emerging technology will mature into a practical, scalable or economically viable system. Media attention, investment activity and promotional claims should not automatically be treated as evidence of technological maturity.
An AMTEF-oriented horizon-scanning process can therefore examine technical maturity, potential applications, infrastructure requirements, economic conditions, regulatory implications, environmental consequences, social acceptance and organizational readiness.
Horizon scanning should also be continuous. Because technological environments change rapidly, a one-time assessment may become outdated. Organizations may therefore establish periodic scanning processes and update technology portfolios as new evidence emerges.
Within AMTEF, horizon scanning provides an early-warning and opportunity-identification mechanism that can feed into technology roadmaps, research priorities, risk assessments and strategic evaluation.
15.28 Technology Roadmapping
Technology roadmapping connects future needs, technological capabilities, strategic objectives and anticipated timelines. A roadmap can show how technologies, infrastructure, skills, research activities and organizational requirements are expected to develop over time.
A technology roadmap may operate at different levels. An organizational roadmap may describe the technologies required to support strategic objectives. A product roadmap may connect future customer requirements with technological development. A sectoral roadmap may examine how an industry could evolve in response to emerging technologies, regulation and market conditions.
Roadmaps can support investment planning because they help organizations identify when particular technologies or capabilities may need to be developed or acquired. They can also support workforce planning by identifying future skills requirements.
Technology roadmaps can be particularly valuable where technologies depend upon complementary infrastructure. An organization adopting advanced AI, for example, may require improvements in data governance, computing infrastructure, cybersecurity, employee capability and organizational policies before the technology can be implemented effectively.
Roadmaps can also identify dependencies and sequencing. Some technologies cannot be implemented until supporting systems have been upgraded. Similarly, regulatory approval may need to occur before commercial deployment.
However, roadmaps should be treated as adaptive planning instruments rather than fixed predictions. Technological development can occur faster or slower than expected, new competitors may emerge, regulations may change and unexpected technological breakthroughs may alter the anticipated pathway.
Regular review is therefore essential. A roadmap should be updated when significant evidence changes the assumptions on which it was based. This makes roadmapping particularly compatible with continuous technology evaluation.
Within AMTEF, technology roadmapping can connect present evaluation with future planning by identifying how technical capability, organizational readiness, investment, skills, infrastructure and governance requirements may evolve over time.
15.29 Benchmarking
Benchmarking compares a technology, system, process or organization with an appropriate reference point. The reference may be an existing internal system, a competitor, an industry standard, a regulatory requirement, a recognized technical specification or another organization with comparable characteristics.
Benchmarking can help evaluators determine whether a technology's performance is relatively strong, weak or typical within a defined context. For example, an organization's data-processing performance may be compared with an established industry benchmark, while an energy system may be compared with similar installations under comparable operating conditions.
Meaningful benchmarking requires equivalent criteria. Comparing technologies using different operating conditions, system boundaries, workloads or performance definitions can produce misleading conclusions. A data center operating under a different climate and workload should not automatically be compared with another facility without considering these differences.
Benchmarking should therefore clearly identify the comparison basis, data sources, measurement periods and assumptions. Researchers should also distinguish between descriptive benchmarking and causal explanation. A technology may perform differently from a benchmark without the benchmark necessarily explaining why.
Internal benchmarking can also be valuable. Organizations may compare departments, sites or business units using the same technology. Such comparisons may reveal differences in implementation quality, training, management practices or resource availability.
External benchmarking can provide broader contextual information but may introduce comparability problems if organizations differ substantially in size, structure, market conditions or regulatory environments.
Benchmarking should therefore be treated as evidence for contextual interpretation rather than as a simple ranking mechanism. A technology that performs below one benchmark may perform well against another, depending on the criteria and conditions selected.
Within AMTEF, benchmarking can support technical, economic, operational and organizational evaluation, provided that comparison conditions are sufficiently transparent and comparable.
15.30 Usability Testing
Usability testing examines how actual users interact with a technology and whether they can perform intended tasks effectively, efficiently and satisfactorily. Usability is particularly important because technical functionality does not guarantee practical usability. A system may contain sophisticated capabilities while remaining difficult for its intended users to understand or operate.
Relevant measures may include task-completion time, error rates, task-success rates, number of assistance requests, learning time, navigation difficulties and user satisfaction. The appropriate measures depend on the purpose and context of the technology.
Usability testing should ideally involve representative users performing realistic tasks. Testing a university learning platform, for example, should involve actual students and faculty performing activities they would normally undertake rather than simply asking them whether they believe the platform is easy to use.
Controlled usability testing can identify specific interface problems, while field-based usability evaluation can reveal problems that arise under normal working conditions. Both approaches may be useful because laboratory conditions may not reproduce interruptions, time pressure, organizational constraints or competing tasks found in real environments.
Usability also includes accessibility. A technology may be easy to use for experienced users but difficult for individuals with disabilities, limited digital literacy or language barriers. Inclusive usability evaluation should therefore consider the diversity of the intended user population.
Usability can also change over time. New users may initially experience difficulties that decrease as they gain experience, while experienced users may develop workarounds that conceal underlying design problems. Longitudinal evaluation can therefore provide useful information about learning and adaptation.
There is also an important relationship between usability and safety. Poorly designed interfaces can increase the likelihood of user error, particularly in healthcare, transportation, industrial and other high-consequence environments. Usability evaluation should therefore be integrated with risk assessment where user error could produce significant consequences.
User satisfaction alone should not be treated as a complete measure of usability. Users may report satisfaction while performing tasks inefficiently, or they may dislike a system while still achieving tasks effectively. Objective task measures and subjective user feedback should therefore be considered together.
Within AMTEF, usability testing connects technical capability with human experience. It helps determine whether a technology that works according to technical specifications can actually be used effectively, safely, inclusively and sustainably by the people for whom it was designed.
15.31 Security Testing
Security testing is an essential component of technology evaluation because modern technological systems are exposed to continually changing threats. Security evaluation can include vulnerability assessment, penetration testing, red-team exercises, threat modelling, security audits, configuration reviews, access-control testing and other forms of controlled security analysis. The objective is not simply to determine whether vulnerabilities exist but to understand how those vulnerabilities could affect the confidentiality, integrity and availability of information and technological services.
Threat modelling provides an important foundation for security evaluation. It examines the assets that need protection, potential threat actors, possible attack pathways, vulnerabilities and consequences. A threat model can help evaluators understand how an attacker might move through a system and which components represent critical points of exposure. This approach is particularly important for interconnected technologies because a vulnerability in one component may create consequences elsewhere in the technological ecosystem.
Penetration testing takes a more practical approach by attempting to identify and, within authorized boundaries, demonstrate exploitable weaknesses under controlled conditions. It can examine applications, networks, authentication mechanisms, APIs, cloud infrastructure and other components. The purpose of authorized penetration testing is to provide evidence about actual exploitable weaknesses so that organizations can prioritize remediation.
Red-team exercises can extend this approach by testing not only technical controls but also organizational responses. A controlled simulated attack may examine whether security personnel detect the activity, whether escalation procedures operate effectively, whether managers understand their responsibilities and whether communication and incident-response processes function as intended. This makes red-team exercises particularly relevant to AMTEF because cybersecurity is both a technical and organizational capability.
Security audits provide another perspective. They may examine whether policies, procedures, access controls, logging, backup arrangements, data-protection requirements and security governance processes are appropriately established and implemented. An organization may therefore have technically secure software but weak governance or inadequate operational controls.
Security testing should also consider the difference between vulnerability and risk. A vulnerability is a weakness, while risk depends upon the likelihood that the weakness will be exploited and the consequences that would result. A technically significant vulnerability in a highly isolated system may present a different risk from a similar vulnerability in a system connected to critical infrastructure.
Security evaluation should be continuous because threats, technologies, configurations and vulnerabilities change over time. Software updates can introduce new vulnerabilities, previously unknown weaknesses can be discovered, threat actors can develop new techniques and organizational changes can create new access pathways. Consequently, a security assessment conducted before deployment cannot be assumed to remain valid indefinitely.
Security testing should also be integrated with privacy, resilience, governance and business-continuity evaluation. A cyber incident may create financial, legal, operational, social and reputational consequences. Within AMTEF, cybersecurity should therefore be treated as a lifecycle dimension of technology evaluation rather than as a one-time technical inspection.
15.32 Ethical Impact Assessment
Ethical impact assessment examines the moral and societal questions that arise from the design, implementation and use of technology. Important considerations may include autonomy, privacy, fairness, dignity, accountability, transparency, human oversight, consent and potential harm. Ethical evaluation becomes increasingly important as technologies gain greater influence over decisions affecting individuals, organizations and communities.
Ethical assessment should occur throughout technology development rather than only after deployment. Early assessment can influence system architecture, data practices, user interfaces, governance arrangements and accountability mechanisms. Identifying ethical concerns during development may make it possible to redesign the technology before problematic features become deeply embedded in technical and organizational systems.
Artificial intelligence provides an important example. Ethical evaluation may examine how data are collected and used, whether models produce systematically different outcomes across groups, whether users can understand important decisions, whether meaningful human oversight exists and who is accountable when an AI-supported decision causes harm. These questions cannot necessarily be answered by measuring model accuracy alone.
For surveillance technologies, privacy and proportionality may require particular attention. Evaluators may need to consider what information is collected, for what purpose, how long it is retained, who can access it and whether the level of surveillance is proportionate to the legitimate objective being pursued.
Ethical evaluation should also examine questions of responsibility. Increasing automation can create situations in which multiple actors contribute to a decision or outcome. Developers, technology vendors, organizational managers, professional users and automated systems may all play different roles. Evaluation should therefore establish where responsibility and accountability reside.
Ethical impact assessment should also consider vulnerable populations. Technologies may have different consequences for individuals depending upon age, disability, digital literacy, economic circumstances, access to infrastructure or other contextual factors. An apparently neutral technological design can therefore produce unequal experiences or outcomes.
Transparency is another important consideration. Stakeholders should, where appropriate, be able to understand the purposes for which technology is used and the principal implications of its operation. Complete technical transparency may not always be possible, particularly with complex systems, but meaningful explanations and accountability mechanisms should be considered.
Within AMTEF, ethical assessment should not be regarded as a final approval stage added after technical evaluation. Ethical considerations should influence problem definition, design, data collection, implementation, monitoring and eventual retirement of technology. This creates an ethical lifecycle approach rather than a one-time compliance exercise.