Chapter 15. Methods and Tools for Technology Evaluation — Part 1 of 4
15.1 Introduction
A multidimensional technology-evaluation framework requires multiple research methods because no single method can adequately capture every dimension of a technology. Technologies are simultaneously technical systems, economic investments, organizational interventions, human experiences, social structures, environmental processes and governance challenges. Each of these dimensions generates different forms of evidence.
Technical performance may require controlled experiments, engineering tests, simulations, benchmarking and reliability analysis. Economic value may require financial modelling, cost-benefit analysis, cost-effectiveness analysis and total-cost-of-ownership analysis. Human impact may require interviews, surveys, observations, usability testing and ethnographic approaches. Environmental impact may require life-cycle assessment, carbon analysis, resource analysis and biodiversity assessment. Risk may require scenario analysis, failure-mode analysis, fault-tree analysis and stress testing. Governance may require legal, regulatory, institutional and policy analysis.
Consequently, technology evaluation is inherently methodological and interdisciplinary. The evaluator must select methods according to the research question rather than selecting a method first and attempting to force every evaluation problem into that methodological structure.
The central methodological principle of AMTEF is therefore:
Evaluation question → Evidence requirement → Appropriate method → Analysis → Integration → Decision
This sequence helps prevent methodological mismatch. A survey, for example, may provide valuable information about user perceptions but cannot by itself establish the technical reliability of an engineering system. Similarly, laboratory performance testing may demonstrate technical capability but cannot establish whether employees will accept the technology or whether its organizational costs are sustainable.
Technology evaluation therefore requires methodological pluralism.
15.2 Evaluation as Research
Technology evaluation should be treated as a structured research process rather than as an informal managerial judgement. A rigorous evaluation begins with a clearly defined problem and proceeds through objectives, questions, criteria, evidence collection, analysis and integration.
The process can be represented as:
Research problem → Evaluation objectives → Research questions → Evaluation criteria → Evidence collection → Analysis → Integration → Decision
The research problem establishes what requires investigation. Evaluation objectives specify what the assessment is intended to determine. Research questions translate those objectives into answerable questions. Evaluation criteria define what constitutes acceptable or desirable performance.
Evidence collection then provides the empirical material required to address the questions. Analysis converts evidence into findings, while integration brings together findings from different dimensions. The final decision should be based upon the integrated evidence rather than on one isolated indicator.
This structure also improves transparency because other researchers, managers, regulators and stakeholders can examine how conclusions were reached.
Evaluation as Research
Figure X.X Evaluation as Research — Conceptual Framework for Treating Technology Evaluation as a Structured Research Process
- RESEARCH PROBLEM
- Evaluation Objectives
- Research Questions
- Evaluation Criteria
- Evidence Collection
- Analysis
- Integration
- DECISION
Framework interpretation: Technology evaluation is conceptualized as a structured research process in which a defined research problem leads to evaluation objectives and research questions. These questions establish the evaluation criteria, guide systematic evidence collection and analysis, and culminate in the integration of findings to support an informed decision.
Source: Developed by the researcher.
15.3 Defining the Evaluation Problem
Before selecting research methods, evaluators should define precisely what is being evaluated, why it is being evaluated, and what decisions the evaluation is expected to support. The term “technology” is broad and may refer to a physical product, software application, digital platform, algorithm, artificial intelligence system, infrastructure network, manufacturing process, medical device, service, organizational technology, or interconnected technological ecosystem. Consequently, the first methodological task is to establish the object of evaluation and its boundaries. Without a sufficiently clear definition, evaluators may collect large quantities of information that are individually valid but collectively fail to answer the central evaluation problem.
Technology should rarely be treated as an isolated object. Most contemporary technologies operate within wider technical, organizational, economic, environmental and social systems. An electric vehicle, for example, is not simply a vehicle. Its evaluation may require consideration of battery technology, charging infrastructure, electricity generation, software, telecommunications, maintenance systems, road infrastructure, user behaviour, manufacturing processes, mineral supply chains and end-of-life recycling. Similarly, an artificial intelligence platform cannot necessarily be evaluated by examining the software model alone. Its performance may depend on the quality and representativeness of its data, computing infrastructure, user behaviour, organizational procedures, cybersecurity controls, human oversight, procurement arrangements, regulatory requirements and governance mechanisms.
Defining the evaluation problem therefore requires the evaluator to establish an appropriate system boundary. A narrow boundary may produce an incomplete assessment because important external effects are excluded. An excessively broad boundary, however, may make the evaluation impractical because almost every technological system interacts with numerous external systems. The evaluator must therefore establish boundaries that are sufficiently comprehensive to capture material consequences while remaining feasible for systematic investigation.
Problem definition should also identify the intended purpose of the technology. A technology cannot be evaluated meaningfully without understanding what it is supposed to accomplish. A hospital information system designed to improve information availability should be evaluated partly in relation to information accuracy, accessibility, interoperability and clinical workflow. An educational technology intended to improve student learning should be assessed in relation to learning outcomes, engagement, accessibility, teaching practice and student experience rather than simply the number of users or system logins.
The distinction between technological capability and technological usefulness is particularly important. A system may possess substantial technical capability but provide limited practical value if it does not address the underlying problem. Similarly, a technically imperfect system may nevertheless produce meaningful benefits in a particular context. The evaluation problem should therefore be expressed in terms of the relationship between the technology, the problem it is intended to address, the context in which it operates and the outcomes that matter to stakeholders.
Problem definition should also identify the principal stakeholders affected by the technology. These may include technology developers, managers, employees, customers, students, patients, citizens, regulators, communities, suppliers and other organizations. Different stakeholders may experience different benefits, costs and risks. A technology that reduces organizational operating costs may simultaneously increase employee workload. A digital service that improves access for some users may create accessibility barriers for others. Consequently, the evaluation problem should not be defined exclusively from the perspective of the technology owner or purchaser.
Another important component is temporal definition. Evaluators should specify whether they are examining expected future performance, current performance, short-term implementation outcomes or long-term consequences. Technologies may perform differently at different stages of their lifecycle. Initial implementation may involve substantial costs and disruption, while benefits may emerge gradually. Conversely, a technology may initially perform well but generate maintenance, cybersecurity, environmental or organizational problems over time.
The evaluation problem should ultimately establish the central question that the research is designed to answer. A useful formulation may be expressed as: What technology is being evaluated, for what purpose, in what context, for which stakeholders, against which criteria, over what period, and with what evidence? This formulation creates the foundation for subsequent evaluation objectives, research questions, methodological choices and evidence requirements.
Within AMTEF, problem definition is therefore not a preliminary administrative activity. It is a foundational analytical stage. If the evaluation problem is poorly defined, subsequent data collection and analysis may be technically rigorous but strategically irrelevant. Clear problem definition improves methodological alignment and helps ensure that the final evaluation addresses the actual decision that organizations, policymakers, researchers or other stakeholders need to make.
15.4 Evaluation Objectives
Once the evaluation problem has been defined, the evaluator should establish explicit evaluation objectives. Evaluation objectives describe what the assessment is intended to determine, understand, measure or explain. They should be sufficiently specific to guide evidence collection while remaining broad enough to reflect the multidimensional nature of modern technology. A single objective such as “determine whether the technology works” is generally inadequate because “works” may involve technical performance, financial value, user acceptance, organizational feasibility, environmental sustainability, security, ethical acceptability and long-term resilience.
An AMTEF-oriented evaluation may therefore seek to determine whether a technology is technically effective, economically viable, organizationally feasible, socially acceptable, environmentally sustainable, secure, ethically responsible and appropriately governed. These objectives do not necessarily have equal importance in every evaluation. Their relevance depends on the characteristics of the technology, the evaluation context and the decision being considered. For example, cybersecurity may be a central objective for a financial information system, whereas biodiversity effects may be particularly important for a large infrastructure project.
Evaluation objectives should also distinguish between immediate outputs and longer-term outcomes. A technology may successfully produce its intended output without achieving the desired outcome. For example, an online learning platform may successfully deliver educational content, but successful content delivery does not necessarily demonstrate improved learning. Similarly, a hospital information system may reduce the time required to enter patient information without necessarily improving patient safety. Evaluation objectives should therefore connect technological activities with meaningful consequences.
Objectives can also relate to different stages of the technology lifecycle. Early-stage evaluation may focus on technical feasibility, proof of concept, resource requirements and major risks. During pilot implementation, the evaluation may emphasize usability, effectiveness, interoperability, organizational readiness and stakeholder acceptance. During mature operation, attention may shift toward long-term reliability, total cost of ownership, environmental sustainability, cybersecurity, resilience, scalability and system-wide consequences. At the end of the technology lifecycle, evaluation may focus on replacement, decommissioning, recycling, data migration and residual environmental or social effects.
Good evaluation objectives should also be measurable or assessable. This does not mean that every objective must be reduced to a numerical indicator. Some objectives are appropriately evaluated through qualitative evidence. For example, an objective concerning user trust may require interviews, focus groups or observational evidence in addition to survey measurements. The important principle is that the objective should be sufficiently clear to identify what constitutes relevant evidence.
Objectives should also be logically connected to evaluation questions. If an objective is to determine whether a technology improves operational efficiency, the evaluator should establish what efficiency means in the specific context and which indicators can demonstrate improvement. If an objective is to understand organizational readiness, the evaluation should examine leadership, skills, infrastructure, resources, culture, governance and change-management capability.
Evaluation objectives should therefore create a chain of methodological alignment extending from the original problem through research questions, criteria, evidence and analysis. In AMTEF, the objectives serve as an important bridge between conceptual evaluation and empirical investigation. They prevent evaluators from collecting information simply because it is available and instead encourage them to collect evidence because it contributes directly to answering an identified evaluation question.
15.5 Quantitative Methods
Quantitative methods provide numerical evidence concerning measurable characteristics, relationships, processes, performance levels or outcomes. They can include experiments, surveys, statistical analysis, performance testing, financial modelling, reliability analysis, cost-benefit analysis, cost-effectiveness analysis, environmental measurement and other forms of numerical assessment. Quantitative methods are particularly valuable when evaluators need to establish the magnitude, frequency, rate, difference or probability associated with a technological phenomenon.
For example, an energy technology may be evaluated through measures such as energy efficiency, electricity output, capacity factor, availability, degradation rate, operating cost and lifecycle emissions. A software system may be assessed using processing time, system availability, transaction volume, error rates, response time, cybersecurity incidents and user activity. A healthcare technology may be assessed through clinical outcomes, sensitivity, specificity, false-positive rates, false-negative rates, treatment duration or other measures appropriate to the technology and clinical context.
One major advantage of quantitative methods is their ability to facilitate comparison. Evaluators may compare a new technology with an existing system, compare different technological configurations, examine changes before and after implementation, or compare outcomes between different user groups. Quantitative data can also support statistical inference when appropriate sampling and research designs are employed.
Experiments can provide particularly strong evidence when the evaluation question concerns whether a technological intervention produces a measurable change under controlled conditions. Performance testing can determine whether a system meets technical specifications. Reliability analysis can examine failure rates and operating lifetimes. Financial models can estimate costs, revenues, savings and investment requirements. Surveys can quantify perceptions such as usefulness, satisfaction, trust, readiness and acceptance.
However, quantitative measurement should not be confused with comprehensive evaluation. The existence of numerical evidence does not guarantee that all relevant dimensions have been captured. A technology may demonstrate high technical reliability while generating significant user frustration. An artificial intelligence system may achieve high predictive accuracy while raising concerns about privacy or fairness. A digital platform may reduce administrative costs while increasing workload for particular groups of employees.
Measurement quality is also critical. Quantitative findings depend on the quality of the variables, instruments, sampling procedures, data sources and analytical techniques used. Poorly defined indicators can create false precision. A numerical result may appear highly authoritative while actually representing an incomplete or inappropriate measurement of the underlying concept.
Quantitative methods should therefore be selected according to the evaluation question rather than because numerical evidence appears more scientific or objective. In AMTEF, quantitative evidence represents one important component of the evidence base, but it is integrated with qualitative, contextual, environmental, ethical, governance and risk evidence when these dimensions are relevant.
15.6 Qualitative Methods
Qualitative methods explore meanings, experiences, perceptions, contextual processes, organizational behaviour, social interactions and interpretations that may not be adequately represented through numerical indicators. Common qualitative methods include interviews, focus groups, observation, case studies, document analysis and thematic analysis. These methods are particularly important when evaluators need to understand how people experience technology and why particular outcomes occur.
Technology implementation is fundamentally a human and organizational process. Users do not always interact with technology according to the assumptions of designers or managers. They may develop workarounds, avoid particular functions, create informal procedures, share accounts, use alternative tools or modify established workflows. Qualitative research can reveal these behaviours and help explain why actual technology use differs from formal system design.
Qualitative evidence is particularly valuable when evaluating organizational, social and ethical dimensions of technology. Interviews can explore employee concerns about workload, autonomy, surveillance, job security or professional identity. Focus groups can reveal collective perceptions and disagreements among stakeholder groups. Observation can identify differences between prescribed procedures and actual practice. Document analysis can reveal organizational assumptions, policies, governance arrangements and historical decisions that influence implementation.
For example, a quantitative survey may show that employees report moderate satisfaction with a newly introduced information system. This result provides useful information, but it may not explain the underlying reasons. Interviews could reveal that employees are generally satisfied with the system's functionality but are concerned about increased workload, insufficient training, poor integration with existing systems or uncertainty concerning future responsibilities.
Qualitative research can therefore contribute explanatory depth. It may identify mechanisms that help explain quantitative findings and can reveal unexpected consequences that were not anticipated when the evaluation was designed. This is particularly important in technology evaluation because technological systems often produce emergent effects when they interact with organizational cultures, professional practices and human decision-making.
The quality of qualitative research depends on appropriate sampling, systematic data collection, reflexivity, transparent analytical procedures and careful interpretation. Qualitative findings should not be treated as informal opinions simply because they are expressed in words rather than numbers. Properly designed qualitative research can produce systematic and analytically valuable evidence.
Within AMTEF, qualitative methods are particularly important for ensuring that technology evaluation remains human-centred. Technical performance describes what a system does; qualitative inquiry can help explain how people experience what the system does and how those experiences influence implementation, acceptance, adaptation and consequences.
15.7 Mixed-Methods Research
Mixed-methods research integrates quantitative and qualitative evidence within a coherent research design. Creswell and Plano Clark (2018) describe mixed methods as an approach in which quantitative and qualitative data are collected and integrated to provide a more comprehensive understanding of a research problem. The central principle is not simply to use two methods but to meaningfully connect different forms of evidence during the research process and interpretation.
Mixed methods are particularly appropriate for AMTEF because technology generates both measurable outcomes and contextual experiences. A technological system may produce observable changes in cost, speed, accuracy, reliability or productivity while simultaneously influencing trust, workload, professional identity, organizational culture, autonomy and stakeholder relationships.
For example, quantitative evidence may establish that system processing time decreased by 30 percent after implementation, while interviews may reveal that employees experienced increased workload because the system transferred certain tasks from administrative staff to professional employees. Both findings may be accurate. The quantitative evidence demonstrates an improvement in one dimension of performance, while the qualitative evidence identifies a potentially important organizational consequence.
Mixed methods therefore encourage evaluators to avoid simplistic conclusions based on a single type of evidence. Numerical improvement does not necessarily mean that the overall implementation has been successful, just as user dissatisfaction does not necessarily demonstrate technical failure. Different forms of evidence may address different dimensions of the same evaluation problem.
Mixed-methods research can also support triangulation. When different sources of evidence converge, confidence in an interpretation may increase. When they diverge, the disagreement itself becomes analytically valuable. For example, system logs may indicate high utilization while interviews reveal that users access the system because they are required to do so and subsequently rely on alternative tools for important tasks. Such divergence should not simply be treated as a methodological problem. It may reveal important characteristics of actual technology use.
Mixed-methods designs can be particularly useful when technical, organizational, human, environmental and social dimensions must be evaluated simultaneously. The evaluator may combine performance data, financial analysis, surveys, interviews, observations, environmental measurements and governance analysis within a single integrated evaluation architecture.
However, mixed methods should not be used merely to make a study appear more comprehensive. The methods must have clear complementary purposes. The evaluator should determine why both forms of evidence are required, how they will be connected and how conflicting findings will be interpreted.
Within AMTEF, mixed-methods research provides a practical methodological foundation for multidimensional technology evaluation. It enables evaluators to connect measurable technological performance with the human, organizational and contextual realities within which the technology operates.
15.8 Sequential Explanatory Design
A sequential explanatory design generally begins with quantitative evaluation followed by qualitative investigation. The quantitative phase identifies patterns, relationships, differences or outcomes, while the qualitative phase is subsequently used to explain, interpret or investigate those findings in greater depth.
For example, a survey might show that only 55 percent of employees accept a newly introduced technology. The numerical finding establishes the extent of acceptance within the population studied, but it does not necessarily explain why acceptance is limited. The subsequent qualitative phase could involve interviews with selected employees to investigate the reasons behind the result.
Interviews might reveal insufficient training, concerns about increased workload, poor system integration, uncertainty about job responsibilities, inadequate technical support or perceptions that the technology does not provide sufficient benefit. These explanations can transform an apparently simple quantitative finding into a more meaningful understanding of the adoption problem.
The qualitative phase may also investigate unexpected quantitative findings. Suppose system-use data show that one organizational department has significantly lower utilization than other departments. Interviews and observation could investigate whether the difference results from infrastructure limitations, workflow incompatibility, managerial practices, professional norms or user concerns.
Sequential explanatory design is particularly valuable when numerical patterns require contextual explanation. It can also support more targeted qualitative sampling because the quantitative phase can identify particular groups, cases or outcomes that warrant further investigation.
The two phases should be explicitly connected. The quantitative results should inform the qualitative research questions, participant selection or areas of investigation. The qualitative findings should then be integrated with the quantitative results during interpretation rather than presented as two unrelated studies.
Within AMTEF, sequential explanatory design is especially useful when evaluators first need to determine what is happening and subsequently need to understand why it is happening. The design therefore supports movement from measurement to explanation.
15.9 Sequential Exploratory Design
Sequential exploratory research reverses the order of investigation. The initial qualitative phase explores experiences, perceptions, processes, concepts or emerging issues, and the subsequent quantitative phase develops measures or tests patterns identified during the qualitative investigation.
This approach can be particularly useful when evaluators do not yet understand the variables that are important to the technology evaluation. In emerging technological fields, existing measurement instruments may not adequately capture the experiences of users or stakeholders. Beginning with qualitative investigation allows the evaluator to discover relevant concepts before attempting to quantify them.
For example, interviews with employees may identify several previously unrecognized barriers to artificial intelligence adoption. Participants may describe concerns relating to transparency, loss of professional autonomy, uncertainty about accountability, lack of explainability, insufficient training and changes to professional roles. Researchers can subsequently develop or adapt a survey instrument to investigate the prevalence and relative importance of these barriers across a larger population.
The qualitative stage can therefore contribute to instrument development. Researchers may use themes emerging from interviews to formulate survey items, define constructs or establish response categories. The subsequent quantitative phase can then determine whether the concepts identified in the smaller qualitative sample appear across a wider population.
Sequential exploratory designs can also be valuable in culturally or contextually distinctive environments where standardized instruments developed elsewhere may not adequately reflect local realities. For example, a technology acceptance instrument developed in a highly digitized environment may not fully capture the infrastructure, institutional or cultural conditions affecting technology adoption in another country or sector.
The transition from qualitative exploration to quantitative measurement must nevertheless be undertaken carefully. Not every theme identified in interviews should automatically become a quantitative variable. Researchers must assess conceptual clarity, relevance, measurability and theoretical coherence before constructing quantitative instruments.
Within AMTEF, sequential exploratory design is particularly useful when the evaluator needs to discover the dimensions of a technology problem before attempting to measure them systematically. It supports methodological development in emerging or poorly understood evaluation contexts.
15.10 Convergent Design
In a convergent mixed-methods design, quantitative and qualitative data are collected during approximately the same period and subsequently integrated during interpretation. The design is useful when different forms of evidence address complementary aspects of the same evaluation problem and when the evaluator wishes to obtain a multidimensional understanding within a defined period.
For example, an organization may simultaneously collect system-performance data and conduct interviews with users. Performance data may indicate that the technology is functioning reliably, with high uptime and low technical error rates, while interviews may reveal significant usability problems. Users may report that the system is technically dependable but difficult to navigate, requires excessive steps for common tasks or does not fit established workflows.
The two datasets should not automatically be treated as contradictory. They may be measuring different dimensions of the technology. Technical reliability concerns whether the system remains operational and performs according to specified technical requirements, whereas usability concerns the quality of interaction between users and the system. A technology can therefore demonstrate strong technical reliability while simultaneously producing poor user experiences.
The strength of the convergent design lies in the integration of these perspectives. The evaluator can compare quantitative and qualitative findings, identify areas of convergence, investigate areas of divergence and develop an integrated interpretation. Where both forms of evidence point toward the same conclusion, the interpretation may gain additional support. Where they differ, the evaluator should investigate the reasons for the difference rather than simply selecting one result as correct.
Convergent design can also incorporate other forms of evidence. For example, an evaluation of an artificial intelligence platform could combine usage statistics, performance testing, user surveys, interviews, cybersecurity assessments, privacy analysis and governance-document review. These forms of evidence can be collected within the same general evaluation period and subsequently integrated through the AMTEF framework.
The integration stage is critical. Merely placing quantitative and qualitative findings in separate chapters does not necessarily constitute effective mixed-methods integration. The evaluator should explicitly examine how the findings relate to one another and what the combined evidence indicates about the technology.
A useful AMTEF principle is therefore that evidence should be integrated at the interpretation stage even when data are collected through different methodological approaches. The objective is not to force all evidence into a single numerical score but to construct a coherent understanding of technological performance, consequences, risks and contextual conditions.
Convergent design is particularly appropriate when decision-makers require a broad assessment within a relatively defined period. It enables technical and numerical evidence to be considered alongside human and contextual evidence, thereby supporting a more comprehensive technology evaluation.
15.11 Experimental Evaluation
Experimental methods allow researchers to examine whether changes in an independent variable are associated with differences in measured outcomes under controlled or systematically structured conditions. In technology evaluation, experimentation can be used to determine whether a technological intervention produces a measurable change in performance, behaviour, efficiency, safety, productivity, learning, health outcomes or another defined outcome. The essential feature of experimental evaluation is that the evaluator deliberately introduces or compares conditions and subsequently observes whether measurable differences occur.
Technology experiments may compare a technology with no technology, an existing system with a new system, one technological configuration with another, or different levels of technological exposure. For example, a university may compare student learning outcomes between classes using a conventional teaching approach and classes using an AI-supported learning platform. An organization may compare processing times before and after introducing an automated workflow. An energy researcher may compare the performance of two different storage configurations under controlled operating conditions.
A basic experimental structure requires careful identification of the independent variable and dependent variable. The independent variable represents the intervention, technological condition or configuration being examined, while the dependent variable represents the outcome being measured. Additional variables may need to be controlled or statistically accounted for because they could influence the observed outcome. In a university technology study, for example, student performance may be influenced not only by the learning platform but also by prior academic achievement, instructor characteristics, course difficulty, attendance and student engagement.
Randomized controlled trials may be appropriate in some healthcare, education and social settings, particularly when ethical and practical conditions permit random assignment. Randomization can reduce systematic differences between groups and strengthen the basis for interpreting observed differences. However, randomization is not automatically appropriate or feasible for every technology evaluation.
Organizations may be unable to randomize departments because a new information system is introduced across the entire organization. Ethical considerations may prevent researchers from withholding a potentially beneficial healthcare intervention from a comparison group. Infrastructure technologies may require system-wide implementation, making individual-level randomization impractical. Network technologies may also produce spillover effects because users in one group interact with users or systems in another group.
Technology experiments may therefore require different experimental structures. Laboratory experiments can provide high levels of control, whereas field experiments can provide greater realism. Controlled technical testing may establish whether a system performs according to specifications, while real-world experimentation can determine whether the technology produces the intended outcome under practical operating conditions. The evaluator must recognize that strong control and high ecological realism may sometimes involve trade-offs.
Experimental evaluation also requires attention to implementation fidelity. If a technology is not implemented as intended, a weak outcome may reflect poor implementation rather than inadequate technological capability. Similarly, a positive result may depend upon unusually high levels of technical support that cannot be sustained after the research period. Evaluators should therefore document how the intervention was implemented, who used it, under what conditions, for how long and with what level of support.
Sample size and statistical power are also important where statistical inference is intended. A small sample may fail to detect a meaningful difference, while a very large sample may detect statistically significant differences that have little practical importance. Consequently, statistical significance should be considered alongside effect size, practical significance, confidence intervals and the substantive importance of the observed change.
Experimental evaluation should therefore be selected according to the nature of the research problem rather than treated as universally superior to other methods. An experiment is particularly valuable when the evaluation question concerns whether a defined intervention produces a measurable change under identifiable conditions. However, many technology questions concern experiences, governance, ethics, organizational adaptation, long-term sustainability or system-wide consequences that cannot be adequately addressed through experimentation alone.
Within AMTEF, experimental evidence can therefore provide strong evidence concerning selected dimensions of technological performance while being integrated with qualitative, economic, environmental, ethical and governance evidence where necessary.
15.12 Quasi-Experimental Evaluation
Quasi-experimental methods can be useful where random assignment is not possible but researchers still seek to estimate changes associated with a technological intervention. They are particularly relevant in organizational, educational, healthcare, public-sector and infrastructure settings where technologies are implemented under naturally occurring conditions rather than through researcher-controlled randomization.
A major advantage of quasi-experimental evaluation is that it can provide a stronger basis for examining changes than simple before-and-after comparisons. A basic pre-implementation and post-implementation comparison may show that performance changed, but the change could have resulted from many other factors occurring during the same period. Quasi-experimental designs attempt to address this problem by introducing comparison groups, repeated observations or statistical controls.
Difference-in-differences analysis, for example, can compare changes over time between an intervention group and a comparison group. Suppose one group of organizational units adopts a new digital platform while another comparable group continues using the existing system. The evaluator can examine whether the change in outcomes in the technology group differs from the change observed in the comparison group.
Interrupted time-series analysis examines observations collected repeatedly over time and determines whether the introduction of a technology is associated with a change in the level or trend of an outcome. This can be particularly useful for technologies implemented across an entire organization or population. For example, researchers might examine monthly processing times before and after implementation of an automated administrative system.
Matched comparisons can also be used to create more comparable groups when naturally occurring differences exist. Organizations, departments, schools or users receiving the technology can be matched with similar units that have not yet received it. Matching does not eliminate all possible sources of bias, but it can improve comparability.
Other quasi-experimental approaches may include regression discontinuity, synthetic-control approaches and controlled interrupted time-series designs where the research context permits. The appropriate method depends on the structure of the available data and the assumptions that can reasonably be defended.
These approaches can strengthen causal inference under appropriate assumptions, although the evaluator must examine whether those assumptions are justified. For example, difference-in-differences designs generally require consideration of whether intervention and comparison groups followed sufficiently similar trends before implementation. If this condition is not reasonably supported, interpretation becomes more difficult.
Quasi-experimental research therefore requires careful attention to alternative explanations. Economic changes, staffing changes, policy changes, seasonal patterns, leadership changes or other technological interventions may occur simultaneously with the intervention being evaluated. The evaluator should document such factors and consider whether they could plausibly explain the observed outcome.
Within AMTEF, quasi-experimental methods are particularly valuable for real-world technology evaluation because many organizational technologies cannot be randomly assigned. They provide a methodological bridge between controlled experimentation and purely observational research.
15.13 Survey Research
Survey research is one of the most widely used methods for evaluating perceptions, attitudes, experiences and self-reported behaviours associated with technology. Surveys can assess user acceptance, satisfaction, perceived usefulness, perceived ease of use, trust, readiness, perceived risk, technology anxiety, digital confidence, organizational support and other measurable constructs.
Survey research is especially useful when an evaluator needs to collect standardized information from a relatively large population. A university implementing an artificial intelligence platform, for example, may survey students and faculty concerning perceived usefulness, ease of use, trust, concerns about academic integrity, privacy expectations and readiness for continued use.
A strong survey begins with clear conceptualization of the constructs being measured. Researchers should define what they mean by terms such as “trust,” “acceptance,” “readiness,” “satisfaction” or “perceived usefulness.” Constructs that are conceptually unclear can produce ambiguous measurement and make subsequent statistical interpretation difficult.
Questionnaire items should be designed to represent the underlying construct appropriately. Researchers should avoid unnecessarily complicated wording, double-barrelled questions, ambiguous terminology and questions that encourage respondents toward a particular answer. The response scale should also be appropriate to the construct being measured.
Validity and reliability are central considerations. Content validity concerns whether the instrument adequately represents the relevant dimensions of the construct. Construct validity concerns whether the instrument actually measures the theoretical concept it is intended to measure. Criterion-related validity may be relevant where survey scores are compared with an external criterion. Researchers may also examine factor structures where appropriate to determine whether items behave consistently with the theoretical model.
Cronbach's alpha may be used as one indicator of internal consistency for multi-item scales, although it should not be treated as a complete measure of scale quality. A high internal-consistency coefficient does not establish that a scale measures the intended construct. Excessively similar items can sometimes produce high internal consistency while adding little conceptual breadth. Researchers should therefore interpret reliability statistics alongside validity evidence, theoretical justification and the characteristics of the instrument.
Sampling is another important consideration. The evaluator should define the target population, determine how participants will be selected and consider the potential for sampling bias. A survey of technology users that excludes individuals who discontinued use may overestimate acceptance. Similarly, surveying only technically confident employees may produce a distorted representation of organizational readiness.
Response bias should also be considered. Participants may provide socially desirable responses, particularly when technology use is associated with managerial expectations or employment evaluation. Nonresponse bias may occur when individuals who do not participate differ systematically from those who respond. Researchers should therefore report relevant response information and acknowledge limitations.
Survey findings can provide valuable quantitative evidence, but they should not automatically be interpreted as direct measures of actual behaviour. A respondent may report that a system is easy to use but rarely use it. Conversely, a respondent may report dissatisfaction while continuing to use the system because it is mandatory. Where important decisions depend on actual behaviour, survey findings can be strengthened through system logs, observation, interviews or other evidence.
Within AMTEF, survey research is therefore most useful when integrated with other evidence sources. Surveys can establish the prevalence and distribution of perceptions across a population, while qualitative and behavioural evidence can help explain those perceptions.
15.14 Interviews
Interviews can provide detailed evidence concerning stakeholder experiences, ethical concerns, organizational barriers, implementation challenges, perceptions of risk and unintended consequences. They are particularly useful when the evaluator needs to understand how participants interpret a technological intervention rather than simply determine how frequently a particular response occurs.
Semi-structured interviews are particularly useful in technology evaluation because they combine consistency with flexibility. Researchers can establish a common set of questions for all participants while allowing participants to explain unexpected experiences and introduce issues that were not anticipated during the initial design of the study.
A technology evaluation interview may explore participants' experiences before implementation, their expectations concerning the technology, their experiences during implementation, changes in their work practices, perceived benefits, difficulties encountered, training received, concerns about risk and their views concerning future use. Such interviews can reveal implementation processes that would otherwise remain invisible.
Interview sampling should reflect the purpose of the evaluation. Where technology affects multiple stakeholder groups, researchers should avoid relying exclusively on one group. A university technology evaluation, for example, may need to consider faculty, students, administrators, technical staff, senior leadership and possibly external stakeholders. Different groups may experience the same technology in substantially different ways.
Interview quality also depends on the relationship between interviewer and participant. Interviewers should create an environment in which participants can provide honest responses without unnecessary fear of organizational consequences. Confidentiality and informed consent are particularly important when participants are discussing organizational failures, privacy concerns, management practices or professional difficulties.
Researchers should also remain aware of interviewer effects. The wording of questions, tone of voice, assumptions and reactions can influence responses. Reflexivity is therefore important, particularly in qualitative technology evaluation where researchers may have prior beliefs concerning the value or risks of a technology.
Analysis should move beyond simply listing participant statements. Researchers should identify recurring patterns, differences between stakeholder groups, relationships among themes and explanations for observed experiences. Where appropriate, thematic analysis can be used to organize interview evidence systematically.
Interview findings can also be integrated with quantitative evidence. If survey results indicate low acceptance, interviews may explain the causes. If system-performance data indicate successful implementation but interviews reveal substantial frustration, the evaluator can investigate why technical success and user experience differ.
Within AMTEF, interviews are therefore an important method for examining the human and organizational dimensions of technology and for ensuring that evaluation does not become limited to technical performance indicators.
15.15 Focus Groups
Focus groups provide an opportunity to examine collective perspectives, interactions and shared experiences. They can be useful for employees, students, patients, customers, citizens, community members and other stakeholder groups affected by technology.
The group setting can stimulate discussion that may not occur during individual interviews. Participants respond to one another, clarify their experiences, challenge assumptions and sometimes identify shared concerns that emerge only through interaction. This can make focus groups particularly useful during exploratory stages of technology evaluation.
For example, a university considering an AI-based assessment platform could conduct separate focus groups with students and faculty. Students might discuss concerns concerning privacy, transparency and assessment fairness, while faculty might focus on workload, academic integrity, assessment redesign and institutional policy. The interaction within each group may reveal areas of agreement as well as disagreement.
However, group dynamics can also influence responses. Participants may hesitate to disagree with senior colleagues, managers or socially dominant individuals. A particularly confident participant may influence the direction of discussion. Participants may also suppress sensitive concerns because they do not wish to discuss them publicly.
Skilled facilitation is therefore important. The facilitator should establish clear ground rules, encourage balanced participation, manage dominant voices and create opportunities for quieter participants to contribute. The composition of the group should also be considered carefully. Combining individuals with substantially different levels of organizational authority may reduce the openness of discussion.
Confidentiality requires particular attention because, unlike individual interviews, researchers cannot completely control what participants may disclose outside the group. Participants should therefore be informed of the limits of confidentiality and encouraged to respect the privacy of other participants.
Focus groups can also be organized according to stakeholder characteristics. Separate groups may be appropriate where participants have substantially different experiences or power relationships. The resulting data can then be compared across groups.
Within AMTEF, focus groups are particularly valuable for understanding collective perceptions, stakeholder expectations and social responses to technology. They can complement individual interviews, surveys and observational evidence.