Chapter 15. Methods and Tools for Technology Evaluation — Part 4 of 4
15.51 Evaluation Across the Technology Lifecycle
Technology evaluation should change as the technology develops because different questions become important at different stages of the technology lifecycle. A method that is appropriate during early research may not be sufficient during large-scale deployment, while a method appropriate for mature operational systems may be unnecessary or impossible during early experimentation.
During research and development, laboratory testing, modelling, simulation, expert assessment and technical feasibility studies may dominate. At this stage, evaluators are primarily concerned with whether the underlying technological concept can function and whether further development is justified. Safety, ethical and environmental considerations should nevertheless begin at this stage rather than being postponed until commercialization.
During prototyping, usability testing, controlled experiments, engineering validation and user feedback become increasingly important. The evaluator can examine whether the prototype performs as intended and whether users can interact with it effectively. Problems identified during this stage may be less costly to correct than problems discovered after large-scale implementation.
During pilot implementation, mixed methods and operational monitoring become more valuable. A pilot can provide evidence about how the technology performs within a real organizational context. It can reveal infrastructure limitations, workflow problems, training needs, user resistance, security weaknesses and unintended consequences.
During scaling, organizational readiness, economic sustainability, cybersecurity, governance, environmental impacts and workforce capability become increasingly important. A technology that performs successfully in one department may encounter different challenges when introduced across an entire organization. Scaling can change the cost structure, workload, risk exposure and social consequences of the system.
During mature operation, longitudinal monitoring, incident analysis, maintenance evaluation, post-implementation review and real-world evidence become central. The evaluator should examine whether the technology continues to achieve its intended objectives and whether changing conditions have created new risks or opportunities.
Eventually, technologies may reach obsolescence, replacement or retirement. Evaluation at this stage should consider migration requirements, data preservation, environmental disposal, cybersecurity risks associated with legacy systems and the costs of transition to alternative technologies.
Technology evaluation should therefore be dynamic rather than static. The evaluation framework should evolve as the technology moves from conception to development, implementation, operation, scaling and eventual retirement.
15.52 Case Study: Evaluating a University AI Platform
Consider a university evaluating an artificial-intelligence platform intended to support teaching, learning and administrative activities. The university may initially be attracted to the platform because of its claimed ability to improve efficiency, personalize learning, automate administrative activities or support students and faculty. However, a comprehensive technology evaluation should extend considerably beyond a demonstration of technical capability.
Quantitative evidence could include student performance, system uptime, platform usage, response time, processing efficiency, support requests, implementation cost, operating cost and other appropriate indicators. Depending on the purpose of the platform, the university might also examine student retention, assessment completion, learning engagement or administrative processing time. These measures should be carefully defined because changes in performance may have multiple explanations.
Quantitative evidence alone, however, would not reveal how students and faculty experience the platform. Qualitative evidence could therefore include student interviews, faculty interviews, administrator perspectives, observations of actual use and analysis of institutional documents. These methods could reveal whether users trust the system, whether faculty understand its capabilities and limitations, whether students understand appropriate use, and whether the technology creates new administrative or academic burdens.
Risk evaluation could include cybersecurity testing, privacy assessment, threat modelling, access-control analysis, incident-response assessment and examination of potential misuse. If the platform processes student information or academic records, data governance becomes particularly important. The institution would need to establish who can access the data, how data are stored, how long they are retained, whether data are transferred to third parties and what contractual safeguards exist.
Academic integrity would represent another important evaluation dimension. The university may need to examine how AI affects assessment design, authorship, plagiarism detection, examination procedures, student disclosure requirements and faculty responsibilities. The objective should not simply be to determine whether AI can generate academic content but to understand how its use changes teaching, assessment and academic practice.
Environmental evaluation could examine energy consumption, computing infrastructure, data-centre requirements and hardware requirements where relevant. Governance evaluation could examine institutional AI policies, procurement arrangements, accountability structures, data governance, academic-integrity procedures, staff training and mechanisms for reviewing system performance.
The evaluation should also examine accessibility and inclusion. A technology may produce benefits for some users while creating barriers for others. Accessibility requirements, language support, disability considerations, digital literacy and unequal access to infrastructure should therefore form part of the evaluation.
The final evaluation would integrate technical, economic, educational, human, social, ethical, environmental, security and governance evidence. A vendor demonstration would constitute only one source of evidence and should not substitute for independent institutional evaluation. The university should distinguish between what the technology is technically capable of doing, what the vendor claims it can do, what users actually do with it and what measurable outcomes it produces within the institution.
15.53 Case Study: Evaluating an Energy-Management System
Consider an organization introducing an intelligent energy-management system designed to reduce electricity consumption, improve operational efficiency and support sustainability objectives. The system may use sensors, automated controls, analytics and artificial intelligence to monitor energy use and adjust equipment operation.
Technical evaluation could measure energy consumption, system accuracy, equipment performance, control responsiveness, reliability and system availability. The organization could establish a baseline before implementation and compare subsequent performance against appropriate reference periods. Seasonal variations, changes in occupancy, production levels and external environmental conditions should be considered because energy consumption may change for reasons unrelated to the technology.
Economic evaluation could compare installation and operating costs with measured savings. Total cost of ownership should be considered rather than focusing solely on the initial purchase price. Software licences, sensor replacement, maintenance, training, cybersecurity, technical support and system upgrades may all affect the long-term economic value of the technology.
Qualitative evaluation could examine employee experiences and identify whether the system changes behaviour or creates additional administrative burdens. Interviews and observation might reveal that employees do not understand automated controls or that building occupants override system recommendations because they perceive them as inconvenient. Such findings could explain why technical performance does not automatically translate into organizational benefits.
Environmental evaluation could estimate changes in energy consumption and associated greenhouse-gas emissions. However, the evaluation should also consider the environmental requirements associated with the sensors, computing infrastructure, replacement equipment and eventual disposal. A broader lifecycle perspective can therefore provide a more complete environmental assessment.
Risk analysis could investigate cybersecurity, system failure, incorrect automated controls and potential operational disruption. Because an intelligent energy-management system may be connected to building-management or industrial-control infrastructure, a cyber incident could potentially have consequences beyond information confidentiality. Resilience and recovery capabilities should therefore be considered.
Organizational evaluation could examine training, leadership support, maintenance capability, governance arrangements and responsibility for system oversight. The organization should establish who is responsible for responding to system alerts, correcting errors and authorizing changes to automated controls.
The integrated evidence would allow the organization to determine not merely whether the system functions technically but whether it produces sustainable value within the organizational context. A successful evaluation would therefore connect technical performance with actual energy savings, financial consequences, employee experience, environmental outcomes, security and organizational capability.
15.54 Common Methodological Errors
Technology evaluation can be weakened by a range of methodological errors. One common problem is evaluating only the dimensions for which data are easiest to obtain. Technical performance data may be readily available from system logs, while social, ethical, organizational and environmental consequences may require more extensive research. If evaluators focus only on readily available data, the resulting evaluation may provide an incomplete representation of the technology.
Another common problem is excessive reliance on vendor-provided evidence without independent verification. Vendor information can be valuable for understanding technical specifications, intended capabilities and operating requirements, but claims should be assessed in relation to independent testing, operational evidence and the specific context in which the technology will be used.
Inappropriate comparison groups can also produce misleading conclusions. For example, comparing a new technology with an outdated system without controlling for differences in workload, infrastructure, users or operating conditions may exaggerate the apparent benefits of the new system. Similarly, comparing two organizations without considering their different institutional contexts can produce misleading conclusions.
Confusing correlation with causation is another important methodological error. If organizational performance improves after a technology is introduced, the improvement cannot automatically be attributed entirely to the technology. Other factors, such as staff training, organizational restructuring, new policies, market conditions or changes in leadership, may have contributed to the outcome. Strong causal claims require research designs capable of supporting them.
Technology evaluators may also confuse subjective perceptions with objective performance. Users may report that a system feels faster or easier to use, but such perceptions should not automatically be treated as evidence of measured technical performance. Conversely, a system may produce objectively strong performance while users experience it as difficult or burdensome. Both forms of evidence may be important, but they answer different questions.
Ignoring missing data is another serious problem. Missing observations may occur because systems fail, users do not respond to surveys, certain groups are excluded or particular events are not recorded. Missing data may therefore contain information about the technology itself. Evaluators should investigate why data are missing rather than automatically treating missing observations as irrelevant.
Failure to document assumptions can also undermine an evaluation. Financial models, environmental calculations, simulations and forecasts frequently depend upon assumptions about future prices, technology lifetime, adoption, utilization, maintenance, energy sources and other variables. If these assumptions are not documented, readers cannot properly assess the basis of the conclusions.
Presenting uncertain projections as established facts is another methodological weakness. Forecasts and scenarios are inherently conditional. They describe possible outcomes under particular assumptions and should not be presented as guaranteed future results.
A further error is methodological overconfidence, in which sophisticated analytical techniques are applied to weak or incomplete data. Advanced statistical models, machine-learning algorithms and complex simulations can create an appearance of scientific precision even when the underlying measurements are poor. Sophisticated analysis cannot compensate for poor sampling, weak constructs, biased data, inappropriate comparison groups or inadequate research design.
Another problem is methodological fragmentation. Separate specialists may evaluate technical, economic, environmental, ethical and organizational dimensions without integrating their findings. This can result in several technically correct analyses that collectively fail to answer the actual decision question. AMTEF therefore emphasizes evidence integration and interpretation across dimensions.
Finally, evaluators should be alert to confirmation bias. An organization may become committed to a technology before the evaluation is completed and then unintentionally emphasize evidence supporting adoption while discounting contradictory findings. Evaluation criteria, data sources, analytical methods and decision rules should therefore be established as transparently as possible before conclusions are finalized.
A rigorous technology evaluation should consequently be characterized by methodological transparency, appropriate evidence selection, explicit assumptions, recognition of limitations, triangulation, uncertainty analysis and alignment between research questions and methods. The purpose of methodology is not to make an evaluation appear more sophisticated; it is to make the resulting evidence sufficiently credible, relevant and useful for the decision being considered.
15.55 Methodological Transparency
A credible technology evaluation should clearly describe the methods used to generate, analyse and interpret evidence. Methodological transparency means that readers, stakeholders and decision-makers should be able to understand what was evaluated, why particular methods were selected, how data were collected, which assumptions were made, how the evidence was analysed and what limitations affect the findings. Transparency is therefore not simply a matter of documenting technical procedures. It is a fundamental component of research credibility and responsible decision-making.
A transparent evaluation should identify the evaluation problem and objectives, define the evaluation questions, explain the criteria used to judge the technology and describe the sources of evidence. It should also specify the population or system being studied, the sampling strategy where relevant, the measurement instruments, data-collection procedures and analytical techniques. Where surveys or interviews are used, the evaluator should explain how participants were selected and how the instruments were developed. Where technical testing is conducted, the evaluation should describe the testing environment, equipment, performance criteria and relevant assumptions.
Transparency is particularly important when different forms of evidence are combined. If an evaluation integrates technical testing, financial modelling, surveys, interviews, life-cycle assessment and risk analysis, the evaluator should explain how these forms of evidence were brought together. Readers should be able to distinguish between empirical findings, modelled estimates, expert judgements, assumptions and interpretations. This distinction becomes particularly important when evidence from one dimension appears to conflict with evidence from another.
The assumptions underlying analytical models should also be explicitly documented. A financial model may depend upon assumptions about inflation, discount rates, technology lifetime, maintenance costs, utilization, energy prices and replacement requirements. An environmental assessment may depend upon assumptions concerning electricity sources, manufacturing processes, transportation distances, recycling rates and equipment lifetimes. A risk model may depend upon assumptions concerning the probability and consequences of particular events. If such assumptions remain hidden, the evaluation may appear more objective than it actually is.
Data provenance is another important component of transparency. Evaluators should identify where important data originated, when they were collected and whether they were independently verified. Where secondary data are used, their relevance to the evaluation context should be explained. Where vendor data are used, the evaluator should distinguish vendor claims from independently generated evidence.
Methodological transparency also requires the reporting of limitations. No technology evaluation is completely free from limitations. Data may be incomplete, samples may be relatively small, some outcomes may be difficult to measure, external conditions may change, and some consequences may not become visible until long after implementation. Acknowledging these limitations does not weaken a study. On the contrary, it allows readers to interpret the findings more appropriately.
Transparency also facilitates independent scrutiny and replication. Other researchers or organizations may be able to examine the methods, challenge assumptions, conduct alternative analyses or reproduce parts of the evaluation. This is particularly valuable when evaluations influence substantial financial investments, public policy, healthcare decisions, educational practices, environmental interventions or infrastructure development.
Methodological transparency is also closely connected to ethical responsibility. If a technology affects individuals or communities, stakeholders should have an appropriate basis for understanding how conclusions concerning safety, effectiveness, privacy, fairness or social impact were reached. Transparent evaluation therefore supports not only scientific quality but also accountability.
An important principle is that transparency should extend to both positive and negative findings. Evaluators should not selectively report evidence that supports a preferred conclusion while omitting evidence that identifies limitations or adverse effects. A balanced evaluation should document contradictory findings, unexpected outcomes and areas where evidence remains inconclusive.
Within AMTEF, methodological transparency can therefore be regarded as a cross-cutting requirement. Every major evaluation dimension should have an identifiable evidence base, methodological rationale, analytical procedure and statement of uncertainty. The objective is to ensure that the final evaluation represents a traceable chain from question to evidence, from evidence to analysis and from analysis to interpretation.
15.56 Integrating Methods Without Creating False Precision
Multidimensional technology evaluation requires the integration of different forms of evidence, but integration should not be confused with numerical aggregation. Not every characteristic of a technology can or should be converted into a numerical score. Some characteristics can be measured with considerable precision, while others involve human experience, ethical judgement, institutional values or contextual interpretation.
For example, technical efficiency may be expressed as a percentage, system availability may be reported as a measurable proportion of operating time, processing speed may be measured in milliseconds, and operating cost may be expressed in monetary terms. These measurements can provide useful quantitative evidence. By contrast, dignity, trust, perceived fairness, ethical acceptability, institutional legitimacy and quality of human relationships may require qualitative evidence and structured professional judgement.
The difference between measurement and judgement should therefore remain visible within the evaluation. A numerical measure may appear highly precise while still representing only one aspect of a broader phenomenon. For example, a system may achieve a very high accuracy rate but still be unsuitable if its errors are concentrated among particular populations. Similarly, a technology may produce measurable cost savings while creating unacceptable privacy or organizational consequences.
Attempting to convert every evaluation dimension into a single numerical score can create false precision. Suppose technical performance receives one score, economic performance another, social impact another and ethical acceptability another, after which all scores are combined into one overall value. The resulting number may appear objective, but it depends upon assumptions about measurement scales, weighting, comparability and trade-offs. Those assumptions may be more contestable than the final number suggests.
AMTEF should therefore be understood as an evidence-integration framework rather than as a mandatory numerical scoring system. Integration means bringing evidence together in a structured way so that relationships, trade-offs, uncertainties and conflicts can be examined. It does not necessarily mean reducing every dimension to the same mathematical scale.
Where numerical aggregation is appropriate, the evaluator should make the assumptions explicit. Multi-criteria decision analysis, for example, may legitimately use weights to represent the relative importance assigned to different criteria. However, the meaning of the resulting score depends upon the validity of the criteria, measurement methods and weights. Sensitivity analysis should therefore be used where appropriate to determine whether different reasonable assumptions produce substantially different conclusions.
A useful approach is to distinguish between measurable indicators and evaluative judgements. Measurable indicators provide empirical information, while evaluative judgements interpret what that information means within a particular decision context. Both are necessary. A technical measurement can show what a system does, but judgement may be required to determine whether that performance is sufficient for the intended purpose.
The integration process should therefore preserve important qualitative information rather than eliminating it. A technology may have strong technical and economic performance but create a significant ethical concern. Instead of allowing a numerical average to conceal that concern, the evaluation should identify the conflict explicitly. This approach supports more informed decision-making because stakeholders can see where the major trade-offs exist.
The central principle is therefore that methodological integration should increase understanding rather than merely increase numerical complexity. A sophisticated evaluation is not necessarily one that contains the greatest number of calculations. It is one that provides an appropriate, transparent and defensible representation of the evidence relevant to the decision.
15.57 Decision Rules and Evidence Thresholds
Different technologies require different levels of evidence before important decisions are made. A low-risk consumer application may be capable of deployment following relatively limited testing, while a technology that influences healthcare decisions, controls critical infrastructure, manages financial transactions or processes sensitive personal information may require considerably stronger evidence, independent review and continuous monitoring.
Evidence thresholds should therefore be proportionate to the potential consequences of failure. The greater the potential harm, uncertainty, scale of exposure and difficulty of reversing a decision, the stronger the justification for robust evaluation. This principle can be described as proportionality in evaluation.
A technology evaluation should establish, where possible, what evidence is required before progression from one stage to another. For example, a research prototype may require evidence of technical feasibility before moving to pilot testing. A pilot may require evidence of safety, usability, organizational readiness and acceptable performance before being scaled. A large-scale deployment may require evidence concerning security, regulatory compliance, economic sustainability and long-term operational performance.
Evidence thresholds can also differ according to the type of risk involved. A technology may have strong evidence of technical reliability but insufficient evidence concerning privacy. Another technology may have extensive user-acceptance evidence but limited evidence concerning cybersecurity. An integrated evaluation should therefore avoid allowing strong evidence in one dimension to automatically compensate for insufficient evidence in another when the dimensions concern materially different risks.
The concept of minimum evidence thresholds can be especially important for high-consequence technologies. Certain conditions may need to be satisfied before deployment regardless of the technology's performance in other areas. For example, an organization may determine that critical security vulnerabilities must be resolved before implementation, even if the technology offers substantial economic benefits.
Decision rules should also distinguish between evidence of absence and absence of evidence. If an evaluation does not identify a particular risk, that does not necessarily mean that the risk does not exist. It may instead mean that the evaluation lacked sufficient evidence to detect it. This distinction is especially important for emerging technologies where long-term evidence may be limited.
Evidence thresholds should also reflect uncertainty. When evidence is limited but potential consequences are substantial, controlled pilots, additional testing, restricted deployment or enhanced monitoring may be appropriate ways of managing uncertainty. The evaluation process can therefore support staged decision-making rather than forcing organizations into an immediate full-scale adoption or rejection decision.
Decision rules should be documented before final conclusions are drawn wherever practicable. Clearly established criteria can reduce the risk that organizations change their standards after seeing the results. They can also improve consistency between evaluations.
The concept of evidence thresholds therefore connects research methodology with governance. Evaluation does not merely produce information; it establishes the evidentiary basis upon which organizations determine whether a technology is sufficiently understood for a particular stage of deployment.
15.58 Continuous Evaluation
Technology evaluation should not necessarily end when an organization decides to adopt a technology. Continuous evaluation is necessary because technology performance, user behaviour, organizational requirements, costs, threats, regulations and external conditions can change over time. A technology that performs effectively at the time of implementation may perform differently several years later.
Software updates can change functionality or introduce new vulnerabilities. User populations may change. Organizational processes may be redesigned. New regulations may alter compliance requirements. Cybersecurity threats may evolve. Suppliers may discontinue components or change pricing structures. Infrastructure may age. New competing technologies may alter the economic or strategic environment.
Continuous evaluation therefore extends the concept of technology assessment from a one-time decision to an ongoing governance process. After implementation, organizations should monitor relevant indicators, investigate significant deviations and periodically reassess whether the technology continues to meet its intended objectives.
The continuous evaluation cycle can be represented as:
Measure → Analyse → Learn → Adjust → Monitor → Re-evaluate
Measurement provides evidence about current performance. Analysis examines patterns, deviations and emerging problems. Learning involves interpreting what has been discovered and identifying implications. Adjustment involves modifying the technology, process, training, governance or implementation strategy where appropriate. Monitoring then determines whether the adjustments have produced the intended result. Re-evaluation provides an opportunity to reconsider the technology as conditions change.
Continuous evaluation is particularly important for technologies based on artificial intelligence and machine learning because model performance can change when data, users or environments change. A model that performs well under one set of conditions may experience degradation when the underlying data distribution changes. Monitoring may therefore need to include model performance, data quality, error patterns, bias indicators and human oversight.
Continuous evaluation is equally relevant to physical technologies. Equipment may experience degradation, maintenance requirements may increase and operating conditions may change. Energy systems, medical technologies, industrial machinery and transportation infrastructure can all require ongoing monitoring.
Post-implementation evaluation can also identify benefits that were not anticipated during the original assessment. Users may discover new applications; organizations may redesign processes around the technology and unexpected efficiencies may emerge. Continuous evaluation therefore does not merely identify problems; it can identify opportunities for improvement and innovation.
An important principle is that monitoring should be linked to predefined escalation mechanisms. If a significant security incident, performance deterioration or safety concern occurs, the organization should know who is responsible for investigating it and what actions may be required. Monitoring without an associated response process can create large amounts of information without effective governance.
Continuous evaluation therefore connects methodology directly with lifecycle governance. It transforms technology evaluation from a static report into an ongoing organizational capability.
15.59 Research Questions
Research into technology-evaluation methodology can investigate how different methods should be selected and integrated when technologies produce simultaneous technical, economic, organizational, environmental, human, social, ethical and governance consequences. One important research question concerns which methodological combinations are most appropriate for different categories of technology and different levels of risk. Researchers can examine whether particular combinations of quantitative, qualitative and analytical methods provide more complete evidence than individual methods used in isolation.
A further area of research concerns mixed-methods integration. Researchers can investigate how quantitative and qualitative findings can be combined without allowing one form of evidence to automatically dominate the other. For example, a study may examine whether strong quantitative performance evidence should be interpreted differently when interviews reveal substantial concerns about usability, workload or trust.
Research can also investigate the representation of uncertainty in technology decisions. Emerging technologies often involve limited historical evidence, uncertain future conditions and rapidly changing environments. Researchers can examine how confidence intervals, sensitivity analysis, scenario analysis, Monte Carlo simulation and expert elicitation can be combined to communicate uncertainty in ways that decision-makers can understand.
Multi-criteria decision analysis provides another important area for investigation. Researchers can examine how MCDA can structure complex technology decisions while avoiding misleading impressions of numerical objectivity. Particular attention can be given to the selection of criteria, weighting procedures, stakeholder participation, sensitivity analysis and the interpretation of composite results.
Further research can investigate how organizations can avoid excessive reliance upon single technology metrics. Many organizations monitor easily measurable indicators because they are convenient, but single indicators can provide an incomplete representation of performance. Research can therefore examine how dashboards and KPI systems can combine technical, financial, human, environmental, security and governance evidence without overwhelming decision-makers.
Triangulation represents another important methodological research area. Researchers can investigate whether combining different methods, data sources, stakeholder groups and time periods improves the credibility, reliability and contextual validity of findings. They can also investigate circumstances in which apparently contradictory findings provide important information rather than representing methodological failure.
Research is also required on lifecycle evaluation. Methods appropriate during research and development may differ from those required during pilot implementation, scaling, mature operation and retirement. Researchers can investigate how evaluation criteria should evolve across these stages and how organizations can maintain continuity of evidence as technologies develop.
Another important research area concerns real-world evidence. Researchers can investigate how operational data, user behaviour, system logs, maintenance records and incident reports can be incorporated into formal evaluation processes. This includes examining how longitudinal evidence can reveal effects that are not visible during short-term trials.
The role of stakeholder participation also warrants continued investigation. Technology decisions often affect groups differently, and researchers can examine how participatory approaches influence the identification of risks, unintended consequences and implementation barriers. The quality of participation, representation of affected groups and influence of stakeholder evidence are particularly important areas for methodological study.
Researchers can further investigate how technology evaluation can support responsible innovation. This includes examining how ethical, social, environmental and governance considerations can be integrated into evaluation before technologies become deeply embedded in organizational systems.
These research questions demonstrate that technology-evaluation methodology is itself an evolving field. As technologies become more interconnected, autonomous and data-driven, evaluation methods must continue to develop in response to new forms of uncertainty, risk and societal impact.
15.60 Chapter Summary
Technology evaluation requires methodological pluralism because different evaluation questions require different forms of evidence. A complex technology cannot normally be understood adequately through one research method because its consequences may extend across technical, economic, organizational, human, social, ethical, environmental, security and governance dimensions.
Quantitative methods provide numerical measurement and are particularly useful for assessing performance, reliability, costs, utilization, outcomes and measurable changes over time. Qualitative methods provide contextual understanding and can reveal user experiences, organizational processes, ethical concerns, implementation barriers and unintended consequences. Mixed-methods research can combine these complementary forms of evidence and provide a more comprehensive understanding of complex technology systems.
Experimental and quasi-experimental methods can assist in investigating effectiveness and, where the design permits, causal relationships. Surveys can examine perceptions, attitudes, acceptance, readiness and satisfaction, while interviews, focus groups and observations can reveal experiences and implementation processes. Case studies provide detailed understanding of technologies operating within real organizational and social contexts.
Economic methods such as cost-benefit analysis, cost-effectiveness analysis and total-cost-of-ownership analysis provide important financial evidence. Multi-criteria decision analysis can structure comparisons across multiple dimensions, particularly where cost alone is insufficient. Life-cycle assessment supports environmental evaluation by examining impacts across stages of a technology's existence. Risk assessment, security testing and ethical-impact assessment address important dimensions of technological consequence that may not be captured through conventional performance measurement.
Sensitivity analysis and Monte Carlo simulation provide tools for examining uncertainty, while confidence intervals and other statistical measures help communicate uncertainty in empirical estimates. Dashboards and KPIs can support ongoing monitoring, provided that indicators are selected according to evaluation objectives rather than simply according to data availability.
Triangulation strengthens evaluation by comparing evidence from different methods, data sources, stakeholder groups and time periods. Importantly, triangulation does not require every source to produce identical findings. Differences between sources may reveal important contextual issues, implementation problems or differences between expected and actual technology use.
Validity and reliability remain essential methodological considerations. A measurement can be consistent without accurately representing the construct that the evaluator intends to measure. Internal validity, external validity, construct validity and ecological validity address different aspects of research quality. Similarly, measurement error, sampling bias, response bias and other sources of systematic or random error must be considered.
Uncertainty analysis is particularly important when evaluating emerging technologies. Technology decisions frequently involve incomplete information concerning future costs, adoption, regulation, technical performance and social consequences. A responsible evaluation should therefore distinguish between what is established, what is estimated and what remains uncertain.
Methodological transparency is also fundamental. Evaluators should document data sources, assumptions, methods, limitations and analytical procedures so that stakeholders can understand how conclusions were reached. This is especially important where technology evaluations influence major financial investments, healthcare decisions, educational practices, infrastructure development or public-interest decisions.
The central AMTEF principle is that methodology should follow the evaluation question. No single method should be assumed to provide a complete assessment of a complex technology. The appropriate methodological design depends upon what needs to be known, the nature of the technology, the context in which it will operate, the consequences of failure, the available evidence and the degree of uncertainty.
AMTEF provides the conceptual architecture, while research methods provide the evidence required to populate that architecture. The framework therefore should not be understood as replacing established research methodologies. Instead, it provides a structure through which different methods can be selected, connected, interpreted and integrated.
The strongest technology evaluations consequently combine methodological breadth with methodological discipline. They use multiple forms of evidence where necessary, but they do not collect information merely for the sake of collecting more information. They identify relevant questions, select appropriate methods, evaluate evidence quality, recognize uncertainty, integrate findings and communicate limitations transparently.
Ultimately, technology evaluation is not simply an exercise in measuring technological performance. It is a structured process for understanding whether a technology is appropriate, sustainable, safe, useful, governable and capable of producing its intended value within a particular context. The quality of that understanding depends heavily upon the quality of the methodology used to produce it.
15.61 Discussion Questions
Why is no single research method generally sufficient for multidimensional technology evaluation? In what ways do technical, economic, organizational, human, social, ethical and environmental evaluation questions require different forms of evidence?
How does the evaluation question influence the selection of research methods? Explain why methodology should follow the evaluation question rather than the evaluator simply selecting a familiar or convenient research method.
What is the value of mixed-methods research in technology evaluation? Discuss how quantitative evidence and qualitative evidence can complement one another and identify circumstances in which one type of evidence may reveal information that the other cannot adequately capture.
Under what circumstances might a sequential explanatory design be particularly useful? Consider situations in which quantitative findings require further explanation through interviews, focus groups, observations or other qualitative methods.
When might a sequential exploratory design provide greater value? Discuss situations in which researchers must first understand stakeholder experiences, organizational processes or emerging issues before developing quantitative instruments or testing broader patterns.
How can triangulation improve the credibility of technology-evaluation findings? Explain how methodological triangulation, data-source triangulation, investigator triangulation and theoretical triangulation can contribute to a more comprehensive evaluation.
What is the difference between internal validity, external validity, construct validity and ecological validity? Explain why each form of validity addresses a different aspect of research quality and why technology evaluations conducted in controlled environments may produce different findings from evaluations conducted in real-world settings.
Why can a reliable measurement instrument nevertheless produce invalid conclusions? Discuss the distinction between consistency and accuracy of measurement and provide examples involving technology acceptance, user satisfaction or system performance.
What are the strengths and limitations of multi-criteria decision analysis? Explain how MCDA can help structure complex technology decisions while also examining the risks associated with criteria selection, weighting, aggregation and assumptions about comparability.
How should evaluators communicate uncertainty to decision-makers? Discuss the role of confidence intervals, sensitivity analysis, scenarios, Monte Carlo simulation, assumptions and qualitative explanations in communicating uncertainty without creating either unjustified confidence or unnecessary confusion.
Why should life-cycle assessment be incorporated into environmental technology evaluation? Consider why evaluating only the operational phase of a technology may overlook environmental impacts associated with raw materials, manufacturing, transportation, maintenance, energy consumption and end-of-life treatment.
How can organizations avoid excessive dependence upon a single technology metric? Discuss why a technology may appear successful when measured through one indicator but produce different results when technical, economic, human, environmental, security and governance indicators are examined together.
What role should qualitative evidence play when quantitative performance appears strong? Consider whether high technical performance automatically establishes usability, trust, organizational acceptance, ethical acceptability or social value.
How can dashboards support technology evaluation without encouraging superficial target chasing? Discuss the importance of selecting meaningful indicators, distinguishing leading and lagging measures, providing contextual information and linking indicators to specific decisions.
Why should evaluation methods change as a technology moves through its lifecycle? Explain how research and development, prototyping, pilot implementation, scaling, mature operation and retirement create different information requirements and methodological priorities.
Why is methodological transparency important in technology evaluation? Discuss how disclosure of assumptions, data sources, analytical procedures and limitations contributes to credibility, accountability and independent scrutiny.
How should evidence thresholds differ between low-risk and high-consequence technologies? Consider how potential harm, uncertainty, scale of exposure, reversibility and societal dependence should influence the amount and quality of evidence required before deployment.
What is the difference between evidence of absence and absence of evidence? Explain why failing to identify a risk does not necessarily demonstrate that the risk does not exist, particularly when evaluating emerging technologies.
How can continuous evaluation contribute to technology governance? Discuss how the cycle of Measure → Analyse → Learn → Adjust → Monitor → Re-evaluate can help organizations respond to changing technology performance, user behaviour, security threats, costs and external conditions.
How can AMTEF integrate different research methods without creating a false impression that complex technological decisions can always be reduced to one numerical score? Discuss the importance of preserving qualitative evidence, documenting assumptions, identifying trade-offs, representing uncertainty and distinguishing empirical measurement from evaluative judgement.
Finally, consider the following broader question: What characteristics should define a rigorous, transparent and responsible technology-evaluation methodology in an environment characterized by rapid technological change, incomplete evidence, interconnected systems and increasing dependence on artificial intelligence and digital infrastructure?
CHAPTER 16