Part IV — Applying Technology Evaluation to Energy, Organizations, Research, Education and Higher Education

Chapter 16. Technology Evaluation in Education and Higher Education — Part 2 of 2

16.22 Technology and Doctoral Education

Doctoral education is a particularly important environment for evaluating educational technology because doctoral students are expected to develop independent research capability and contribute new knowledge. Technology can provide substantial assistance throughout the doctoral journey, from identifying literature and organizing sources to supporting coding, data analysis, language improvement, research planning and document management.

AI-assisted literature exploration, for example, can help researchers identify concepts, terminology and potentially relevant areas of scholarship. Coding assistants may support programming and data-analysis tasks. Language technologies can help researchers whose first language is not English improve grammar, clarity and readability. Research-management systems can assist with organizing large collections of documents and maintaining research records.

However, doctoral education is not simply about producing a completed dissertation. The doctoral process is intended to develop independent scholarly judgement, methodological competence, critical thinking, theoretical understanding and the ability to make defensible research decisions. Excessive dependence on technology could therefore weaken the very capabilities that doctoral education is intended to develop.

A doctoral researcher may use AI to suggest possible research questions, for example, but should be able to explain why a particular research question is appropriate. A researcher may use AI to assist with statistical programming, but should understand the analytical method, assumptions and interpretation of the results. Similarly, AI may improve language, but the researcher must retain ownership of the intellectual argument.

The distinction between assistance and substitution is therefore particularly important. Technology can assist a doctoral researcher without replacing the researcher's intellectual responsibility. The more consequential the scholarly decision, the greater the need for human understanding, verification and accountability.

Technology also changes the role of the doctoral supervisor. Supervisors may increasingly spend less time correcting basic language or formatting problems and more time helping researchers evaluate methodological choices, theoretical arguments, evidence quality, research integrity and the appropriate use of AI. This does not reduce the importance of supervision; instead, it can change the nature of supervisory work toward deeper scholarly evaluation.

Doctoral programmes should therefore establish explicit expectations concerning responsible technology use. Students should understand when AI assistance is acceptable, when disclosure is required, how references must be verified, how confidential research material should be protected and which elements of the doctoral research must remain demonstrably their own intellectual work.

16.23 Technology and Doctoral Supervision

Technology has significantly expanded the possibilities for doctoral supervision. Supervisors and doctoral researchers can communicate asynchronously, exchange documents, conduct online meetings, monitor progress, provide feedback and collaborate on research materials without being physically located in the same place. This is particularly important for international doctoral education and for researchers who cannot regularly attend a university campus.

Asynchronous supervision can allow doctoral researchers to submit drafts and receive structured feedback without requiring both parties to be available simultaneously. Document-collaboration systems can provide version histories and facilitate iterative development. Digital research-management tools can support the organization of references, datasets, instruments and research documentation.

Technology can also support progress monitoring. Universities may use digital systems to track milestones, ethics approvals, proposal development, data collection, chapter submission, examination preparation and other doctoral requirements. Such systems can improve visibility of progress, but they should not become substitutes for meaningful academic dialogue.

One potential risk is the reduction of supervision to administrative monitoring. A doctoral student may technically submit documents on time while still experiencing substantial conceptual or methodological difficulties. Progress indicators therefore need to be interpreted within the wider supervisory relationship.

Human academic judgement remains essential because doctoral research frequently involves ambiguity, originality and intellectual uncertainty. A supervisor may need to challenge an argument, question a methodological decision, identify an unexplored assumption or encourage a student to reconsider an interpretation. These activities require scholarly interaction rather than simple technological processing.

Technology should consequently be evaluated according to whether it strengthens the supervisory relationship rather than merely increasing communication volume. The relevant question is whether digital tools improve the quality, continuity, accessibility and effectiveness of doctoral supervision while preserving intellectual independence, academic responsibility and meaningful human engagement.

16.24 Technology Evaluation in Universities

Because university technologies affect multiple dimensions of institutional activity, technology evaluation should generally be multidisciplinary. A university technology may simultaneously affect teaching, research, administration, finance, cybersecurity, privacy, accessibility, quality assurance, student experience and institutional reputation. No single professional group is likely to possess all the knowledge required to evaluate these dimensions adequately.

Academic leadership can provide strategic alignment by determining whether the proposed technology supports institutional objectives. Faculty members can evaluate pedagogical and disciplinary implications. Students can provide evidence concerning usability, accessibility and actual learning experience. Information technology specialists can assess infrastructure, integration, reliability and technical performance. Information-security specialists can examine cybersecurity risks, vulnerabilities and data-protection arrangements.

Legal and compliance personnel may evaluate contractual obligations, intellectual property, privacy requirements and regulatory considerations. Libraries can assess implications for scholarly communication, information access, research databases and information literacy. Research offices can consider research governance, ethics and research-integrity implications. Quality-assurance professionals can examine alignment with academic standards and institutional processes, while accessibility specialists can assess whether the technology can be used effectively by people with different accessibility requirements.

This multidisciplinary approach reflects the fundamental principle of AMTEF: technology evaluation is multidimensional. A technically reliable system may still be unsuitable if it creates unacceptable privacy risks. A pedagogically useful system may still be unsustainable if its total cost is excessive. An inexpensive system may still be inappropriate if it creates substantial cybersecurity vulnerabilities.

Universities should therefore establish an evaluation process that begins with the institutional or educational problem rather than with the technology itself. The evaluation should identify the intended outcomes, relevant stakeholders, evidence requirements, risks, alternatives and conditions for responsible implementation.

The evaluation process should also include post-implementation review. Approval of a technology should not be considered the end of evaluation. Actual usage, performance, user experience, incidents, costs, learning outcomes and unintended consequences may differ from expectations established during procurement. Continuous evaluation allows universities to determine whether the technology is delivering the expected value and whether modifications are necessary.

16.25 University Technology Evaluation Matrix

A university technology evaluation matrix provides a structured way of examining a technology from multiple higher-education perspectives. The matrix should not be interpreted as a requirement to assign an identical numerical score to every dimension. Some dimensions can be measured quantitatively, whereas others require qualitative judgement, documentary analysis, stakeholder consultation or professional review. The purpose is to ensure that important questions are not overlooked.

The technical dimension asks whether the technology functions reliably under actual university conditions. Evaluation can include system availability, performance, scalability, interoperability, usability, security, maintenance requirements and technical support. The central question is whether the technology performs its intended functions consistently and safely.

The educational dimension asks whether the technology contributes to meaningful learning. This requires examination of learning outcomes, student engagement, instructional quality, assessment, feedback and the relationship between the technology and the pedagogical model. High usage does not automatically demonstrate educational effectiveness. A technology can be heavily used without producing improved learning outcomes.

The research dimension asks whether the technology supports scholarship responsibly. Universities should consider whether the technology improves research processes, supports data analysis, assists scholarly communication or contributes to research productivity while protecting research integrity, confidentiality and methodological quality.

The economic dimension concerns financial sustainability. Evaluation should extend beyond the purchase price to include implementation, integration, training, licensing, maintenance, support, cybersecurity, upgrades, data migration and eventual replacement. Total cost of ownership provides a more comprehensive basis for economic evaluation than initial acquisition cost alone.

The faculty dimension concerns whether academic staff can use the technology effectively and meaningfully. This includes technical competence, pedagogical competence, workload, training, incentives, support and professional autonomy. A system that is theoretically capable but practically difficult for faculty to use may produce limited institutional value.

The student dimension concerns usability, accessibility, educational usefulness and student experience. Universities should examine whether students can access the technology, understand how to use it and derive meaningful educational value from it.

The ethical dimension concerns whether the technology is consistent with academic values such as autonomy, fairness, dignity, accountability, academic freedom and responsible scholarship. Ethical evaluation is particularly important when technologies make predictions, automate decisions or process sensitive information.

The privacy dimension concerns whether student, faculty and research data are adequately protected. Evaluation should consider what data are collected, why they are collected, who can access them, how long they are retained, where they are stored and whether they are shared with third parties.

The equity dimension examines whether the technology reduces or increases inequality. Universities should consider differences in device access, connectivity, disability, language, socioeconomic circumstances and geographical location. An innovation should not be regarded as successful solely because its average institutional outcomes improve if significant groups experience new or intensified barriers.

The governance dimension concerns accountability, policies, responsibilities and institutional oversight. Universities should establish who owns the system, who is responsible for monitoring it, who can make decisions concerning its use and what procedures apply when the technology fails or produces harmful outcomes.

The environmental dimension considers the resource footprint of the technology. Depending on the system, this may include energy consumption, data-centre requirements, hardware production, equipment replacement, electronic waste and other lifecycle effects.

The resulting matrix can therefore be expressed conceptually as follows:

This matrix can be integrated with the wider AMTEF framework to create an institutional decision process. Universities may use evidence from technical testing, financial analysis, educational research, stakeholder consultation, security assessment, privacy review, accessibility testing and environmental assessment. The resulting evaluation should provide a balanced evidence base for adoption, modification, staged implementation or reconsideration of the technology.

16.26 Case Study: Generative AI in Doctoral Education

Consider a university that permits doctoral students to use generative artificial intelligence under clearly defined academic conditions. The institution recognizes that doctoral researchers may use AI for language support, research brainstorming, coding assistance, literature exploration and organizational tasks. At the same time, the university is concerned about inaccurate references, fabricated information, intellectual-property issues, confidentiality and excessive dependence on automated systems.

The first stage of evaluation should identify the educational purpose. If AI is being used to improve language clarity, for example, the university should distinguish language assistance from the generation of the intellectual argument. If AI is being used for coding assistance, the doctoral researcher should still demonstrate an understanding of the analytical procedure and be capable of explaining the code and interpreting the results.

The second stage should consider research-integrity requirements. Researchers remain responsible for the accuracy of their work regardless of whether an AI system was involved in producing part of it. AI-generated references should therefore be independently checked against authoritative sources. Claims, quotations, statistical interpretations and methodological recommendations should likewise be verified.

Confidentiality represents another important evaluation dimension. Doctoral researchers may possess unpublished research, participant information, proprietary organizational data or other confidential material. Such information should not be entered into unauthorized AI systems. Universities should establish clear rules concerning approved systems, data classification and permitted uses.

Intellectual property should also be addressed. Researchers need to understand the relevant institutional, contractual, publisher and legal requirements associated with AI-assisted work. This is particularly important where doctoral research is connected to commercial organizations, funded research or collaborative projects.

The university may therefore establish several responsible-use conditions. AI use should be disclosed where institutional or publication requirements call for disclosure. Researchers should remain responsible for the accuracy and integrity of submitted work. Confidential material should not be entered into unauthorized systems. AI-generated references should be verified independently. Most importantly, substantive scholarly decisions concerning research questions, theoretical interpretation, methodology, evidence evaluation and conclusions should remain the responsibility of the doctoral researcher.

The purpose of such a framework is not simply to prohibit or encourage artificial intelligence. Instead, it establishes conditions under which technology can be used while preserving the educational purpose of doctoral education. The central principle is responsible human oversight. AI may function as a research assistant or productivity tool, but it does not replace the doctoral researcher's scholarly responsibility.

The case also illustrates the value of AMTEF because the decision cannot be resolved through technical performance alone. The university must simultaneously consider educational purpose, research quality, ethics, privacy, intellectual property, student development, institutional governance and academic integrity.

16.27 Case Study: AI-Based Student Early-Warning System

Consider a university that develops an artificial-intelligence-based early-warning system intended to identify students who may be at risk of withdrawing from their studies. The system analyzes available institutional information and generates risk indicators that may be used by student-support professionals to determine whether additional assistance should be offered.

The potential educational benefit is the possibility of earlier intervention. Students experiencing academic difficulty may be identified before problems become more severe, allowing support services to contact them, understand their circumstances and offer appropriate assistance. Such a system could potentially help universities allocate limited support resources more effectively.

However, predictive accuracy alone is insufficient for evaluating such a system. A prediction is not an established fact about an individual student. A student identified as high risk may successfully complete a programme, while a student classified as low risk may subsequently experience serious difficulties. The distinction between prediction and reality must therefore remain central to the evaluation.

Privacy is another important consideration. An early-warning system may process attendance information, academic performance, learning-platform activity, demographic information and other student-related data. Universities must establish what information is necessary, how it is collected, who can access it, how long it is retained and how students are informed about its use.

Bias and unequal outcomes must also be considered. Historical institutional data may reflect existing inequalities or differences in participation. If these patterns are incorporated into a predictive model without appropriate scrutiny, the system could reproduce or amplify existing disparities. Evaluation should therefore examine performance across relevant student groups rather than relying solely on aggregate accuracy.

The institutional response to a prediction is also important. A prediction should not automatically trigger disciplinary action or restrictive decisions. An appropriate use may be to initiate a supportive conversation with a student, allowing the student to explain circumstances that may not be represented in the data. Human interpretation is therefore essential.

Students and student-support staff should be included in the evaluation process. Students can provide insight into how predictive interventions are experienced, including whether they feel supported or unfairly categorized. Support staff can evaluate whether predictions are practically useful and whether the system generates excessive false alerts or inappropriate recommendations.

The evaluation should therefore consider technical performance, educational usefulness, privacy, fairness, transparency, explainability, institutional capacity, human oversight and student experience. The system should be understood as a decision-support technology rather than an autonomous decision-maker.

This case demonstrates a broader principle of technology evaluation in higher education: the value of an AI system depends not only on what the algorithm predicts but also on how the institution interprets the prediction, how people respond to it and what consequences follow for students. Responsible implementation therefore requires the integration of technical evidence with educational, ethical, organizational and human judgement.

16.28 Case Study: Automated Essay Assessment

Automated essay assessment represents an important application of artificial intelligence in education because it attempts to use computational systems to evaluate written student work. A university may employ an AI-based system to provide preliminary assessment, identify characteristics of student writing, generate formative feedback, identify areas requiring attention, or assist academic staff in managing large volumes of written assignments. Such systems may be particularly attractive in institutions with large student populations where faculty members face significant assessment workloads.

One potential advantage is the speed of feedback. An automated system may process a large number of essays considerably faster than human assessors, potentially allowing students to receive preliminary feedback soon after submission. Rapid feedback can be educationally useful when it enables students to identify weaknesses and revise their work while the learning activity remains current. However, speed should not be treated as synonymous with educational quality. Rapid feedback that is inaccurate, generic or unrelated to the learning objectives may have limited value.

Consistency is another potential advantage. An automated system can apply the same programmed criteria repeatedly and may reduce certain forms of variation between individual assessors. Nevertheless, consistency does not necessarily mean validity. A system can consistently apply an inappropriate criterion or systematically reproduce a particular bias. Evaluation must therefore distinguish between consistency, reliability and validity.

Scalability is particularly relevant to large universities and online programmes. An automated system may allow preliminary feedback to be provided to thousands of students without requiring a proportional increase in marking time. This may support formative assessment and provide academic staff with additional information. However, scalability also increases the consequences of systematic errors. If an automated system incorrectly evaluates a small proportion of essays but is used across tens of thousands of submissions, the aggregate effect may become significant.

Bias represents an important risk. Automated assessment systems may perform differently depending on language, writing style, disciplinary conventions, cultural expression, vocabulary, sentence structure or the characteristics of the training data used to develop the system. A system designed primarily around one form of academic English, for example, may evaluate multilingual student writing differently from writing produced by students whose linguistic backgrounds align more closely with the system's underlying data.

Disciplinary variation presents another challenge. Academic writing does not have identical characteristics across all disciplines. A reflective essay in education, a theoretical argument in philosophy, a literature review in business, a methodological discussion in social science and a technical report in engineering may have substantially different conventions. An automated system that evaluates all writing using a single generic model may therefore fail to recognize discipline-specific standards.

There is also a risk that automated systems may reward features that are relatively easy to detect computationally rather than characteristics that genuinely represent scholarly quality. For example, vocabulary complexity, sentence length or structural patterns may be measurable without necessarily demonstrating originality, critical thinking, methodological understanding or theoretical insight. The evaluation of higher-order intellectual capabilities therefore remains a significant challenge.

Student appeals are another important consideration. If an automated system contributes to a grade or assessment decision, students should have access to appropriate procedures for questioning or appealing the outcome. Transparency becomes particularly important when students do not understand how an automated evaluation was generated. Institutions should establish clear procedures identifying who is responsible for reviewing disputed results.

The distinction between formative and summative assessment is therefore essential. Automated systems may be more appropriate for preliminary feedback, low-stakes practice and instructional support than for high-stakes decisions concerning progression, graduation or professional qualification. The greater the consequences of an assessment decision, the stronger the requirements for validity, transparency, human oversight and procedural fairness should become.

For high-stakes assessment, meaningful human oversight should remain central. Human assessors should not simply approve an automated result without examination. They should have sufficient authority, time and expertise to review cases, identify anomalies and override inappropriate automated recommendations. Technology should support academic judgement rather than eliminate it.

An AMTEF evaluation of automated essay assessment should consequently consider technical performance, educational validity, disciplinary appropriateness, reliability, accessibility, language effects, student experience, privacy, ethics, governance, cost and human oversight. The fundamental question is not whether the system can produce a score, but whether its use contributes to valid and educationally meaningful assessment.

16.29 Quality Assurance

Technology adoption should form an integral component of institutional quality assurance rather than being treated solely as an information-technology decision. Universities are responsible for ensuring that their programmes, teaching practices, assessments, student-support systems and research activities meet appropriate academic standards. When technology changes any of these areas, the implications for quality must be considered systematically.

The first quality-assurance question concerns learning outcomes. Universities should determine whether the technology supports the knowledge, skills, competencies and other outcomes specified for the programme or course. A technology may increase interaction, automate administrative activity or generate large quantities of educational content without necessarily contributing to the intended learning outcomes.

Academic standards must also be protected. Technology should not result in a reduction of intellectual expectations merely because certain tasks have become easier to automate. Conversely, technology may enable universities to introduce more authentic, complex and professionally relevant forms of assessment. Quality assurance should therefore examine whether technology changes academic standards and whether those changes remain appropriate.

Assessment validity is another important consideration. If students can use AI or other technologies to complete an assessment, universities should determine whether the assessment still measures the intended capability. This may require assessment redesign rather than simply introducing technological surveillance. The fundamental issue is whether the evidence collected from students remains an appropriate representation of their learning.

Student support must also be considered. Technology adoption may create new support requirements involving technical assistance, academic guidance, accessibility, digital literacy and AI literacy. Quality assurance should therefore examine whether students receive adequate preparation and support.

Staff readiness is equally important. Faculty members need sufficient knowledge, training, time and institutional support to use technology effectively. Introducing a system without preparing academic staff can undermine educational quality even when the technology itself functions reliably.

Risk monitoring should continue after implementation. Quality assurance should examine incidents, student complaints, assessment outcomes, system failures, accessibility problems, cybersecurity events, privacy concerns and unintended educational consequences. This transforms technology evaluation from a one-time procurement activity into a continuous quality process.

Technology should therefore become part of academic-quality governance. Programme reviews, course evaluations, assessment reviews, institutional audits and quality-improvement processes should consider the role of technology where relevant. This approach recognizes that educational technology is not external to academic quality; it increasingly forms part of the environment in which academic quality is produced.

16.30 Accreditation and Regulation

Universities operate within multiple regulatory and quality frameworks. Depending on their jurisdiction and institutional mission, they may be subject to national higher-education regulations, professional accreditation requirements, institutional policies, international quality frameworks, data-protection obligations and standards associated with particular academic or professional programmes.

Technology adoption should therefore be evaluated against the requirements that apply to the institution and the specific activity in which the technology will be used. A technology introduced into a professional programme may have different regulatory implications from one used for routine administrative communication. Similarly, technology used in research involving human participants may be subject to requirements that do not apply to ordinary classroom applications.

Accreditation bodies may require evidence concerning learning outcomes, assessment validity, faculty qualifications, student support, academic resources, governance and quality assurance. Where technology becomes part of these processes, universities should be able to demonstrate that its use does not undermine applicable academic standards.

Regulatory requirements may also concern privacy, cybersecurity, accessibility, records management, consumer protection, intellectual property and data transfer. Cloud-based and AI-enabled technologies can introduce additional complexity because information may be processed by third-party providers or stored across different jurisdictions.

International and transnational education creates further challenges. A university operating programmes across several countries may need to consider both the regulations of its home jurisdiction and requirements applicable in the locations where students are studying or where services are delivered. Technology evaluation should therefore identify the relevant regulatory environment before implementation.

Accreditation and regulatory compliance should not be understood merely as administrative obstacles. They can provide important safeguards for students, researchers, staff and institutions. Technology evaluation should consequently establish whether the proposed system supports compliance, creates new obligations or introduces areas of uncertainty requiring additional review.

16.31 Technology Procurement

Procurement represents a critical stage in technology evaluation because decisions made during procurement can influence institutional performance for many years. Once a university signs a major technology contract, changing systems may become costly and disruptive. Procurement should therefore be based on evidence rather than demonstrations, promotional material or short-term enthusiasm.

Functionality is an obvious consideration. The university must determine whether the technology actually meets the identified educational, research or administrative requirements. Functional requirements should be established before evaluating competing products so that the institution does not allow vendor demonstrations to redefine the problem.

Security should be assessed systematically. Universities hold valuable academic, financial, research and personal information and therefore represent important targets for cyber threats. Procurement should examine authentication, authorization, encryption, vulnerability management, incident response, security testing, access controls and the provider's approach to security updates.

Privacy requires similar attention. Universities should understand what data the system collects, why it collects the data, where information is stored, how it is processed, who can access it and under what circumstances it may be shared. Contracts should clearly establish responsibilities concerning data protection and breach management.

Accessibility should also be incorporated into procurement requirements. A system that cannot be used effectively by students or staff with different accessibility requirements may create educational and legal difficulties. Accessibility should therefore be evaluated before purchase rather than treated as a problem to be solved after implementation.

Interoperability is another major consideration. Universities rarely operate a single technology system. Learning-management systems, student-information systems, library platforms, research-management systems, financial systems and authentication services often need to exchange information. Poor interoperability can create duplicated work, inconsistent data and additional administrative costs.

Data ownership and portability are particularly important in cloud-based environments. Universities should establish who owns institutional data, what rights the vendor has over that data, how data can be exported and what happens to information when a contract ends.

Financial sustainability should include the full lifecycle cost. Licensing fees may represent only one component of expenditure. Universities may also incur implementation costs, integration costs, training expenses, support costs, cybersecurity expenditure, migration costs, upgrade fees and eventual replacement costs.

Support arrangements should also be evaluated. A technically sophisticated system may be difficult to operate without responsive technical support. Universities should establish service-level expectations, escalation procedures, response times and responsibilities for resolving failures.

Vendor demonstrations can provide useful information, but they should not substitute for independent evaluation. Universities should test technologies under realistic conditions, consult users, examine documentation, conduct security and accessibility assessments, review contractual terms and where appropriate conduct pilot implementations. Procurement should therefore be treated as an evidence-based evaluation process rather than simply a purchasing exercise.

16.32 Vendor Lock-In in Higher Education

Vendor lock-in occurs when an institution becomes sufficiently dependent on a technology provider that changing to an alternative system becomes difficult, expensive or operationally disruptive. Higher-education institutions can become vulnerable to lock-in when they rely heavily on proprietary platforms, proprietary data formats, specialized integrations or long-term contractual arrangements.

One consequence can be rising costs. A university that has invested heavily in one system may find it difficult to negotiate or respond to price increases because migration would itself require substantial expenditure. Dependence may therefore reduce the institution's ability to change providers.

Migration difficulty represents another risk. Large universities may accumulate years of student records, course materials, research information, workflows and integrations within a particular platform. Moving this information to another system can require extensive technical and administrative work.

Proprietary data formats can create additional barriers. If institutional data cannot easily be exported in standardized and usable formats, the university may become dependent upon the original provider for continued access. Data portability should therefore be considered during procurement rather than only when an institution decides to leave a vendor.

Loss of institutional control is a broader concern. Universities should retain appropriate control over their data, policies, academic processes and technology architecture. Dependence on a vendor should not result in the university being unable to modify important processes or obtain its own institutional information.

Exit planning is therefore an essential element of responsible procurement. Contracts should address data export, migration assistance, deletion of institutional information where appropriate, documentation, transition support and responsibilities at the end of the relationship. Universities should periodically test whether they could realistically migrate away from a major platform.

Interoperability and open standards can reduce some forms of lock-in. Universities should consider whether systems can communicate with alternative platforms and whether data can be transferred using commonly supported formats. Technology evaluation should consequently consider not only how well a system performs today but also how much institutional flexibility it preserves for the future.

16.33 Technology Governance Committee

A university may establish a Technology Evaluation and Governance Committee to provide multidisciplinary oversight of significant technology decisions. Such a committee can help ensure that technology proposals are examined from technical, educational, research, ethical, legal, financial and organizational perspectives.

Academic leadership can provide strategic direction and determine whether proposed technologies align with institutional priorities. Information-technology representatives can examine architecture, integration, performance and technical support. Faculty representatives can assess educational and disciplinary implications, while student representatives can contribute evidence concerning usability, accessibility and student experience.

Research representatives can examine implications for research processes, research integrity and research data. Library representatives can consider scholarly information systems and access to knowledge resources. Legal and compliance specialists can examine contractual, regulatory, privacy and intellectual-property issues. Quality-assurance representatives can determine whether technology affects academic standards, while cybersecurity specialists can assess security risks. Accessibility specialists can evaluate whether systems can be used by diverse users.

The committee's responsibilities may include evaluating high-impact technologies, reviewing proposals for major technology adoption, approving or recommending pilot programmes, assessing risks, reviewing evidence, monitoring implementation outcomes and recommending policy changes. It may also establish criteria determining which technologies require formal review.

Not every technology requires the same level of governance. A minor software update may require routine administrative approval, whereas an AI system that makes decisions affecting student progression may require extensive multidisciplinary evaluation. Governance should therefore be proportionate to risk, scale, impact, reversibility and sensitivity.

The committee can also support continuous evaluation. After implementation, it may review performance indicators, user feedback, incidents, privacy concerns, security events, accessibility issues and educational outcomes. This allows the institution to move from one-time technology approval toward lifecycle governance.

16.34 Technology Evaluation Policy

A university technology evaluation policy provides the institutional framework through which technology decisions are made consistently. Without such a policy, technology adoption may become fragmented, with individual departments selecting systems according to different criteria and without adequate consideration of institutional risks or long-term consequences.

The purpose of the policy should be clearly defined. It should explain why technology is evaluated and establish the relationship between technology evaluation, educational quality, research integrity, institutional strategy, risk management and responsible innovation.

The scope should identify which technologies require evaluation. This may include learning platforms, artificial-intelligence systems, research technologies, student-information systems, automated decision systems, surveillance technologies, cybersecurity tools, cloud services and major administrative platforms. The policy can establish different review requirements according to the risk and institutional significance of the technology.

The criteria should specify the dimensions that are examined. The AMTEF framework can provide the conceptual basis for examining technical, economic, human, environmental, social, ethical, organizational, governance and risk dimensions. Not every technology will require equal attention to every dimension, but the policy should establish a mechanism for determining which dimensions are relevant.

Responsibility should be clearly allocated. The policy should identify who prepares an evaluation, who reviews it, who approves adoption and who is responsible for monitoring the technology after implementation. Clear responsibility prevents technology governance from becoming a situation in which everyone is involved but no individual or body is accountable.

Evidence requirements should also be established. Depending on the technology, evidence may include technical testing, pilot results, user research, financial analysis, privacy assessments, security assessments, accessibility testing, educational evidence, vendor documentation and independent evidence.

Monitoring requirements should establish how performance is evaluated after adoption. A technology that passed an initial assessment may perform differently when used at institutional scale. The policy should therefore require appropriate post-implementation monitoring.

Review requirements should establish when reassessment is necessary. Major changes in functionality, changes in vendor ownership, cybersecurity incidents, significant increases in cost, new regulatory requirements or evidence of unexpected educational effects may trigger reassessment.

A mature technology evaluation policy therefore establishes a complete lifecycle: identification, assessment, approval, implementation, monitoring, review and, where appropriate, retirement. Technology governance becomes a continuing institutional process rather than a single purchasing decision.

16.35 Technology Evaluation in Teaching

Technology used in teaching should begin with the learning objective rather than the technological capability. Faculty members should first identify what students are expected to know, understand or demonstrate and then determine whether technology contributes meaningfully to those outcomes.

Pedagogical value is therefore central. A technology should provide a clear educational purpose, such as improving feedback, supporting collaboration, enabling simulation, increasing access to resources, facilitating practice or allowing students to engage with authentic professional environments. Novelty alone is insufficient justification for adoption.

Accessibility should be considered from the beginning. Teachers should determine whether students can access the system using appropriate devices and assistive technologies and whether alternative forms of participation are available when necessary.

Reliability is also important because technical failure can interrupt teaching, assessment and communication. Faculty should understand the limitations of the technology and have reasonable contingency arrangements for significant failures.

Student experience provides another important source of evidence. Students can identify usability problems, confusing interfaces, excessive workload, accessibility barriers and differences between the intended and actual use of a system. Student feedback should therefore be combined with objective evidence concerning learning outcomes and participation.

Assessment implications require particular attention. The introduction of AI and other digital technologies may change what students can reasonably be expected to demonstrate through traditional assignments. Faculty may need to redesign assessment so that it measures the intended learning outcomes in the technological environment in which students actually operate.

Technology should therefore be evaluated as a component of pedagogy rather than as an independent product. The central question is whether it strengthens teaching and learning in a sustainable, accessible and academically appropriate manner.

16.36 Technology Evaluation in Research

Research technologies should be evaluated according to their effects on research validity, reproducibility, data governance, ethics, confidentiality and intellectual property. Researchers increasingly depend on software, artificial intelligence, cloud computing, digital databases, statistical packages, automated laboratory systems and other technologies. The quality of research can therefore become partly dependent upon the reliability and appropriate use of these systems.

Research validity requires researchers to understand how technology affects the generation, processing and interpretation of evidence. An automated analytical procedure may produce results efficiently, but researchers must understand its assumptions and limitations sufficiently to determine whether the results are scientifically or academically defensible.

Reproducibility is also important. Researchers should maintain appropriate records concerning software versions, analytical procedures, datasets, transformations, code and other relevant research processes. Where technology contributes substantially to an analysis, the research record should allow appropriate verification or reproduction of the work.

Data governance concerns the collection, storage, processing, sharing and destruction of research information. Researchers must consider access controls, data security, retention requirements, participant confidentiality and institutional policies.

Ethical considerations become particularly important when technology processes information concerning human participants. The use of AI, automated analysis or external cloud services should be consistent with applicable research-ethics requirements and approved research procedures where relevant.

Confidentiality must also be protected. Researchers may handle commercially sensitive information, unpublished findings, personal data or proprietary datasets. Technology selection should therefore take account of where information is processed and who may have access to it.

Intellectual property represents another important consideration. Research outputs may involve patents, copyrighted material, proprietary algorithms, datasets or contractual rights. Technology use should not unintentionally compromise institutional or researcher rights.

Technology evaluation in research should therefore protect both efficiency and scholarly integrity. The objective is not simply to determine whether a tool makes research faster, but whether it enables researchers to produce reliable, transparent, ethical and defensible knowledge.

16.37 Technology Evaluation in Administration

Administrative technologies are increasingly central to university operations. Student admissions, registration, finance, human resources, scheduling, procurement, records management, communication, advising and reporting may all depend upon digital systems. These technologies can improve efficiency and service quality, but their consequences extend beyond simple automation.

Efficiency concerns whether the technology reduces unnecessary processing time, duplication and administrative workload while maintaining appropriate quality. A system that processes applications rapidly but generates large numbers of errors may not produce genuine efficiency.

Accuracy is equally important. Administrative systems may contain information that affects student progression, financial transactions, academic records, staff employment and institutional reporting. Errors in such systems can have significant consequences. Evaluation should therefore examine data quality, validation procedures, error handling and human review.

Data protection is a central consideration because administrative systems often process personal and financial information. Universities should establish appropriate access controls, retention arrangements, security measures and procedures for responding to incidents.

The effect on staff should also be evaluated. Automation may reduce repetitive activities, but it can also change responsibilities, create new monitoring tasks or require staff to manage exceptions generated by automated systems. Staff consultation can therefore reveal effects that may not be visible through technical testing.

Service quality provides another important dimension. Administrative technology should be assessed according to whether it improves the experience of students, faculty and other stakeholders. Faster processing is valuable only if the resulting service remains understandable, accessible and responsive.

Human oversight is particularly important when administrative systems influence consequential decisions. Automated workflows may determine which applications receive attention, which students are contacted or which cases are escalated. Institutions should establish mechanisms for reviewing unusual or disputed cases.

Administrative technology should therefore be evaluated as part of the university's wider service system. The objective is to improve institutional efficiency while preserving accuracy, privacy, accessibility, accountability and appropriate human judgement. AMTEF provides a useful basis for ensuring that administrative technology is assessed not merely as software but as an institutional intervention with technical, organizational, human, economic, ethical and governance consequences.

16.38 Technology Evaluation in Student Services

Student-facing technologies should be evaluated for:

accessibility;

usability;

privacy;

responsiveness;

inclusiveness.

16.39 Higher-Education AMTEF Model

The AMTEF dimensions can be translated into university-specific questions:

Technical: Does the system work?

Educational: Does it improve learning?

Economic: Is it financially sustainable?

Organizational: Can the university support it?

Human: Does it strengthen academic and student agency?

Social: Does it support inclusion?

Ethical: Does it respect academic values?

Environmental: Is resource use responsible?

Risk: Are security and safety risks controlled?

Governance: Is institutional accountability established?

16.40 Research Questions

How should universities evaluate emerging technologies?

How does organizational readiness influence educational technology adoption?

What governance structures are appropriate for generative AI in higher education?

How should universities evaluate AI-enabled assessment?

How can technology support doctoral education without undermining independent scholarship?

What criteria should universities use when procuring AI and digital technologies?

16.41 Chapter Summary

Higher education is both a major adopter and evaluator of emerging technologies.

Technology can expand access, improve administrative efficiency, support learning and transform research.

However, technology adoption can also create risks involving privacy, academic integrity, equity, dependence and educational quality.

AMTEF provides universities with a structured approach for evaluating technology before, during and after adoption.

16.42 Discussion Questions

What makes technology evaluation in higher education distinctive?

How should universities evaluate generative AI?

What is the role of human oversight in AI-based assessment?

How should digital inequality influence educational technology adoption?

How can technology support doctoral education while preserving research independence?