Part III — Evaluating Risk, Intelligent Technologies, Digital Systems, Biotechnology and Human-Centred Technologies

Chapter 10. Evaluating Artificial Intelligence and Intelligent Technologies — Part 2 of 3

10.16 Human Oversight

Human oversight should be meaningful rather than symbolic.

If a human is nominally responsible but simply accepts every AI recommendation, human oversight may not provide effective control.

The evaluator should therefore examine:

whether humans understand the system;

whether they can challenge outputs;

whether they have authority to override;

whether sufficient time exists for review;

whether accountability is clear; and

whether users are trained to recognize system limitations.

UNESCO places human oversight within a broader human-rights-centred approach to AI ethics (UNESCO, 2021).

Human oversight is particularly important where AI systems influence decisions with significant consequences for individuals or organizations. However, simply placing a person somewhere in the workflow does not automatically create effective oversight.

A human reviewer must have sufficient knowledge, authority, time, information, and independence to exercise meaningful judgment. If the reviewer receives hundreds of AI recommendations within a short period, for example, practical review may become little more than confirmation of the system's outputs. This phenomenon can be associated with automation bias, in which users place excessive confidence in automated recommendations.

Effective oversight should therefore be evaluated as a socio-technical control mechanism. The evaluator should examine what information the human receives, how uncertainty is communicated, whether explanations are available, whether alternative evidence can be considered, whether the person can override the system, whether overrides are recorded, and whether there are consequences for inappropriate system use.

The distribution of authority is also important. A human may technically possess an override function but lack organizational authority to exercise it. Alternatively, a human may have authority but lack sufficient technical understanding to determine when an AI recommendation should be rejected.

Human oversight should also be proportionate to risk. Low-risk applications may require periodic review, whereas high-risk systems may require real-time review, multiple levels of authorization, mandatory human confirmation, or the ability to suspend automated operation.

Training is another important component. Users should understand the intended purpose of the AI system, known limitations, uncertainty, common failure patterns, appropriate escalation procedures, and circumstances in which the system should not be used.

Human oversight must also be supported by organizational culture. If employees are pressured to accept automated recommendations because of efficiency targets, cost reductions, or managerial expectations, formal oversight mechanisms may become ineffective.

The evaluation should therefore examine the complete sequence:

AI output → human interpretation → human verification → challenge or acceptance → decision → accountability → feedback.

Human oversight is effective only when the human role adds meaningful judgment rather than merely providing a formal signature to an automated decision.

10.17 AI and Privacy

AI can create privacy concerns through:

extensive data collection;

inference;

facial recognition;

profiling;

data retention;

model memorization;

secondary data use.

Privacy evaluation must therefore consider both direct collection and indirect inference.

The privacy implications of AI extend beyond the traditional question of whether personal information was explicitly collected. AI systems can combine datasets, identify patterns, infer characteristics, generate profiles, and derive information that individuals may never have directly disclosed.

Inference creates a particularly important challenge. A system may not explicitly collect a sensitive characteristic while nevertheless estimating or predicting it from other variables. Consequently, privacy evaluation should consider not only what information enters the system but also what information can potentially be derived from it.

Data minimization should therefore be examined. Organizations should consider whether the data being collected are necessary for the stated purpose or whether substantially more information is being retained than is required.

Purpose limitation is similarly important. Data collected for one purpose may later be reused for another purpose, particularly when AI systems make it technically easy to combine datasets. Such secondary uses should be assessed in relation to legitimate purpose, stakeholder expectations, applicable requirements, and potential harms.

Retention is another dimension. Data that are no longer required may nevertheless remain in databases, backups, logs, model-development environments, or third-party systems. Privacy evaluation should therefore follow the complete data lifecycle.

AI models themselves can also become part of the privacy assessment. Training processes, retrieval systems, prompts, logs, model outputs, embeddings, and other representations may contain or expose information associated with individuals.

Access control must therefore be evaluated alongside data collection. The question is not simply whether the organization possesses personal information, but who can access it, under what circumstances, for what purpose, and with what monitoring.

Privacy should also be evaluated in relation to accuracy and fairness. Incorrect data can result in incorrect profiling or inappropriate decisions, while excessive data collection can create additional risks even when the system performs accurately.

A privacy-aware AI evaluation should therefore follow the information lifecycle:

Data collection → storage → classification → processing → model development → inference → output → sharing → retention → correction → deletion.

The central question is whether the AI system uses information in a manner that is necessary, proportionate, secure, understandable, and appropriately governed throughout its lifecycle.

10.18 AI Security

AI systems can be attacked through:

adversarial examples;

data poisoning;

model extraction;

prompt injection;

malicious inputs;

unauthorized access;

supply-chain compromise.

Goodfellow et al. (2015) demonstrated the vulnerability of machine-learning systems to adversarial examples, illustrating why traditional cybersecurity alone may not capture all AI-specific threats.

AI security therefore requires an extension of conventional cybersecurity rather than its complete replacement. Traditional controls such as identity management, authentication, authorization, encryption, patch management, network protection, logging, monitoring, backup, and incident response remain essential. However, AI introduces additional attack surfaces associated with data, models, prompts, inference processes, training pipelines, retrieval systems, model interfaces, and machine-learning-specific behaviour.

Adversarial examples illustrate one such challenge. Small changes to an input may sometimes cause a model to produce an incorrect classification or prediction while the input remains apparently normal to a human observer. In safety-critical applications, this can create significant consequences.

Data poisoning represents another threat. If malicious or manipulated data enter the training process, the resulting model may learn undesirable behaviour. Evaluation should therefore examine data provenance, validation procedures, access controls, dataset integrity, and mechanisms for detecting anomalous training data.

Model extraction concerns attempt to reproduce or approximate the behaviour of a model through repeated queries. This may create intellectual-property, security, and competitive risks while potentially exposing information about the model's behaviour.

Prompt injection is particularly relevant to generative AI and systems that interact with external information or tools. Malicious instructions may attempt to manipulate system behaviour, override intended constraints, reveal protected information, or cause unauthorized actions.

Supply-chain security is equally important. AI systems frequently depend on external models, libraries, datasets, APIs, cloud infrastructure, pretrained components, and third-party services. A vulnerability in any of these dependencies can affect the overall system.

Security evaluation should therefore examine the complete AI architecture rather than focusing only on the model.

A useful security sequence is:

Threat identification → attack-surface analysis → preventive controls → detection → response → recovery → learning → redesign.

AI security should also be evaluated continuously because threat patterns change as attackers learn about system behaviour. A model that appears secure during development may become vulnerable after deployment because attackers have greater access to its interface and outputs.

The appropriate evaluation question is therefore not whether an AI model is secure in isolation, but whether the complete AI system can resist, detect, contain, recover from, and learn from relevant attacks across its lifecycle.

10.19 Model Drift

Models can deteriorate when the environment changes.

Examples include:

customer behaviour changing;

economic conditions changing;

new fraud patterns;

disease patterns changing;

language changing;

data sources changing.

Model evaluation must therefore continue after deployment.

Model drift occurs when the relationship between the data used by a model and the environment in which the model operates changes over time. A model may therefore remain technically unchanged while its practical performance declines.

Data drift can occur when the statistical characteristics of input variables change. For example, customer demographics, purchasing behaviour, communication patterns, or transaction characteristics may change.

Concept drift occurs when the relationship between inputs and the outcome changes. A fraud-detection system may encounter new methods of fraud, meaning that patterns previously associated with legitimate behaviour may no longer provide the same predictive value.

Label drift can occur when the prevalence or distribution of outcome categories changes. Changes in disease prevalence, economic conditions, customer behaviour, or regulatory environments may alter the distribution of outcomes.

Operational drift may arise from changes in the system surrounding the model. A new database, software update, business process, sensor, data provider, or external API may alter the information supplied to the model.

The evaluation of drift should therefore examine both model performance and the environment in which the model operates. Monitoring only the model's internal technical indicators may fail to identify changes occurring in the wider socio-technical system.

Drift evaluation should include changes in input distributions, output distributions, error rates, subgroup performance, data quality, calibration, operational conditions, and relevant external conditions.

Thresholds should also be established for intervention. Minor changes may simply require continued observation. More significant changes may require recalibration, retraining, additional validation, temporary suspension, or complete replacement of the model.

Retraining itself should not automatically be assumed to solve drift. New data may contain new forms of bias, quality problems, security risks, or distributional changes. Every significant model update should therefore undergo appropriate validation before deployment.

Model drift demonstrates why AI evaluation cannot be treated as a one-time certification activity. Evaluation should be understood as an ongoing process of measurement, interpretation, intervention, and learning.

10.20 AI Lifecycle Monitoring

Monitoring should include:

accuracy;

fairness;

data quality;

system availability;

security;

user complaints;

unexpected outputs;

drift;

incidents.

Monitoring should also establish thresholds for intervention.

AI monitoring is the operational mechanism through which organizations determine whether a system continues to behave within its intended boundaries. It connects initial evaluation with post-deployment governance.

Technical performance monitoring may include accuracy, precision, recall, calibration, latency, availability, error rates, resource utilization, and other measures appropriate to the application.

However, technical performance alone is insufficient. A model may maintain high accuracy while producing increasing disparities across groups, generating unacceptable privacy risks, creating unexpected operational consequences, or being used outside its approved purpose.

Monitoring should therefore include multiple dimensions simultaneously. Changes in data quality may provide an early warning of future performance deterioration. Changes in fairness indicators may indicate emerging disparities. Security events may indicate attempted manipulation. User complaints may reveal problems that technical metrics fail to capture.

Unexpected outputs are particularly important in generative AI. Organizations should establish processes for recording unusual, harmful, misleading, or inappropriate outputs and determining whether they indicate isolated incidents or systematic problems.

Monitoring should also be connected to predefined intervention thresholds. For example, an organization may determine that a substantial increase in false-negative rates requires investigation, that a specific fairness indicator crossing a predefined threshold triggers review, or that a serious security incident requires immediate suspension.

The monitoring system itself should be evaluated. Poor monitoring can create false reassurance. An organization may believe that a system is being controlled because dashboards exist, while the indicators being monitored do not actually capture the most important risks.

Monitoring should also generate evidence for governance. Reports should be available to appropriate decision-makers and should support decisions concerning continued operation, modification, retraining, restriction, suspension, or retirement.

The monitoring lifecycle can therefore be represented as:

Measure → compare with thresholds → identify deviation → investigate → determine significance → intervene → document → reassess → continue monitoring.

Monitoring should also support organizational learning. Repeated incidents, user complaints, unexpected outputs, or drift patterns may indicate that the original assumptions underlying the system were incomplete. In such circumstances, the appropriate response may involve redesigning the broader process rather than simply adjusting the model.

AI monitoring should consequently be understood as a continuous assurance mechanism rather than a technical dashboard.

10.21 AI Governance

AI governance should define:

who approves AI use;

who owns the model;

who validates performance;

who monitors risk;

who handles incidents;

who approves updates;

who can suspend the system.

ISO/IEC 42001 establishes an AI management-system approach for organizations seeking systematic governance of AI activities, while ISO/IEC 23894 provides guidance on AI-specific risk management (ISO/IEC, 2023a, 2023b). ISO/IEC 23894 specifically addresses the integration of AI risk management into organizational AI-related activities.

AI governance provides the organizational structure within which technical evaluation becomes meaningful. A technically well-performing AI system may still create unacceptable organizational risk if responsibility is unclear, decision authority is fragmented, monitoring is weak, or there is no mechanism for intervention.

Governance should begin with purpose. The organization should be able to explain why the AI system exists, what organizational objective it supports, what decisions it influences, and what activities fall outside its approved purpose.

Ownership should then be established. The model owner should not necessarily be the same person as the technical developer. Depending on the organization, there may be separate business ownership, technical ownership, data ownership, risk ownership, compliance responsibility, and operational responsibility.

Validation should also be independent where appropriate. The person or team responsible for developing a model may not be the only party responsible for determining whether it is sufficiently reliable for deployment. Independent review can provide additional assurance, particularly for high-impact applications.

Governance should establish approval requirements before deployment. The organization should determine what evidence is necessary before an AI system can move from experimentation to operational use.

Risk classification is another important governance activity. Different AI applications may require different levels of oversight depending on their potential consequences, degree of autonomy, affected population, sensitivity of data, reversibility of decisions, and exposure to external threats.

Change governance is equally important. AI systems can change through model retraining, software updates, data changes, prompt modifications, configuration changes, vendor changes, and changes in surrounding business processes. Significant changes should therefore trigger appropriate reassessment.

Incident governance should establish who must be notified, who has authority to suspend the system, how incidents are investigated, how affected stakeholders are informed, and how lessons are incorporated into future controls.

Governance should also address system retirement. An organization should know when an AI system should be discontinued, what alternative process will replace it, how historical records will be retained, how access will be removed, and how dependent systems will be transitioned.

AI governance should not be treated as a purely administrative activity. It should connect organizational objectives with technical evidence, ethical principles, legal obligations, risk management, human oversight, cybersecurity, privacy, environmental considerations, and stakeholder interests.

A comprehensive governance structure can therefore be represented as:

Purpose → classification → ownership → risk assessment → validation → approval → deployment → monitoring → incident management → change control → review → suspension or retirement.

The governance question is ultimately broader than whether an AI system works. It is whether the organization has established the authority, responsibility, evidence, controls, and oversight necessary to use the system appropriately.

AI governance should also distinguish between compliance and responsible governance. Compliance generally asks whether applicable legal, regulatory, contractual, or organizational requirements are being met. Governance asks the broader questions of who makes decisions, according to which principles, using what evidence, under what oversight, with what accountability, and with what mechanisms for correction.

This distinction becomes increasingly important as AI systems become embedded in organizational processes. Responsibility can otherwise become fragmented among model developers, data providers, cloud-service providers, software vendors, organizational managers, operational users, and senior decision-makers. Effective governance must therefore make these relationships explicit.

The evaluation of AI governance should consequently examine whether responsibility is identifiable, whether decision authority is clear, whether evidence is available, whether risks are actively monitored, whether stakeholders have appropriate channels for raising concerns, whether significant changes trigger reassessment, and whether the organization can intervene when the system no longer performs within acceptable boundaries.

In this sense, AI governance is not a final layer added after technical development. It is an organizational framework that should operate throughout the entire AI lifecycle, from problem definition and procurement through development, deployment, monitoring, modification, incident response, and retirement.

10.22 AI Risk Management

NIST AI RMF organizes AI risk management around four core functions:

Govern;

Map;

Measure; and

Manage.

These functions provide a useful complement to AMTEF because they focus specifically on AI risk management while AMTEF addresses the broader multidimensional evaluation of technology.

The frameworks should therefore not be treated as competing models.

AMTEF provides the multidimensional evaluation architecture.

NIST AI RMF provides an AI risk-management structure.

ISO/IEC 42001 provides an organizational management-system perspective.

OECD and UNESCO provide broader normative and policy principles.

AI risk management should therefore be understood as one component of a wider technology-evaluation architecture rather than as a substitute for comprehensive technology evaluation. An AI system may have manageable cybersecurity risk while still creating significant privacy, environmental, social, economic, ethical, organizational, or human consequences. Conversely, an AI system may generate substantial benefits while requiring specific controls to manage foreseeable risks. The purpose of AI risk management is therefore not necessarily to eliminate all risk, because complete elimination may be impossible or may prevent legitimate innovation, but to identify, understand, assess, control, monitor, communicate, and continually reassess risks in relation to the intended purpose of the system.

The Govern function establishes the organizational structures within which AI risk management takes place. Governance determines who has responsibility for AI systems, who has authority to approve their use, how risks are escalated, what policies apply, how accountability is established, and how decisions are documented. Governance should begin before model development because important risks can arise from decisions concerning whether an AI system should be developed or procured in the first place.

Governance should also establish organizational expectations concerning acceptable risk. Not every AI application requires the same level of control. An AI system used to recommend entertainment content has a substantially different consequence profile from a system supporting medical diagnosis, critical infrastructure, financial decisions, employment decisions, education, public administration, or safety-critical operations. AI governance should therefore be proportionate to potential consequences.

The Map function focuses on understanding the context in which the AI system operates. This includes identifying the intended purpose, stakeholders, affected populations, system boundaries, assumptions, dependencies, possible harms, benefits, and environmental conditions. Mapping is particularly important because AI risks frequently arise from interactions between the model and its surrounding socio-technical environment.

A model may operate differently depending on who uses it, what information is supplied, how outputs are interpreted, what decisions are made from those outputs, and what other systems are connected to it. Consequently, AI risk mapping should extend beyond the model itself.

Mapping should identify the complete chain:

Problem → data → model → application → user → organizational process → affected stakeholders → external environment.

This approach is consistent with the broader AMTEF principle that technology should be evaluated as a system rather than as an isolated artefact.

The Map function should also identify intended and unintended uses. An AI system may have been developed for one purpose but subsequently used for another. Such function expansion can create risks that were not considered during development. For example, a model initially developed for internal analysis may later be incorporated into customer-facing decisions, automated workflows, or external products.

Third-party dependencies should also be mapped. AI systems may depend on external datasets, pretrained models, cloud infrastructure, application programming interfaces, software libraries, cybersecurity services, identity systems, telecommunications infrastructure, and human service providers. A failure or change in one dependency may affect the entire AI system.

The Measure function concerns the generation of evidence about AI risks and system performance. Measurement should be multidimensional. Technical performance is important, but it should be considered alongside fairness, privacy, security, reliability, explainability, robustness, safety, environmental implications, human factors, and organizational consequences.

Measurement should be based on clearly defined indicators. Where possible, indicators should be connected to the intended purpose and consequence profile of the system. An AI system designed to identify rare but dangerous events may require different measures from a system designed to generate marketing text.

Measurement should also include uncertainty. AI performance is rarely represented adequately by a single average value. Performance may differ across populations, geographical contexts, languages, operating conditions, time periods, and data environments. Consequently, evaluation should identify where evidence is strong, where evidence is limited, and where assumptions remain uncertain.

Validation should also distinguish between laboratory performance and operational performance. A model may achieve strong results under controlled test conditions but behave differently after deployment. Real-world users may supply different inputs, external conditions may change, and integration with other systems may introduce additional failure modes.

The Measure function should therefore incorporate both pre-deployment and post-deployment evidence. Testing before deployment provides evidence about expected behaviour. Monitoring after deployment provides evidence about actual behaviour. These two forms of evidence should be connected through an ongoing evaluation cycle.

The Manage function concerns decisions and actions taken in response to identified risks. Risk management may involve modifying the model, changing data practices, adding human oversight, restricting system functionality, strengthening cybersecurity, improving documentation, changing thresholds, introducing additional validation, limiting deployment, or discontinuing the system.

Risk management should therefore be connected to explicit decision thresholds. The organization should know what level of risk requires routine monitoring, enhanced controls, management review, escalation, temporary suspension, or system retirement.

A central principle is proportionality. A high-consequence system should generally require stronger evidence and more robust controls than a low-consequence system. However, proportionality should not be interpreted simply as a technical risk score. Consequence severity, affected populations, reversibility, uncertainty, exposure, detectability, duration, and available alternatives should all be considered.

AI risk management should also account for benefits. Risk evaluation that considers only potential harms may provide an incomplete basis for decision-making. The evaluator should consider the intended benefits of the system, who receives those benefits, whether benefits are distributed equitably, whether benefits can be achieved through lower-risk alternatives, and whether risks are proportionate to the intended value.

This creates an important connection between AMTEF and AI risk management. AMTEF can provide the broader multidimensional structure within which AI risks and benefits are examined.

A conceptual integration can therefore be represented as:

AI purpose → context → stakeholders → risks and benefits → measurement → controls → deployment → monitoring → intervention → learning → reassessment.

The four NIST functions should also be understood as iterative rather than strictly sequential. Governance influences mapping. Mapping informs measurement. Measurement provides evidence for management. Management produces information that can modify governance, mapping, and measurement.

AI risk management is therefore a continuous organizational capability rather than a one-time compliance exercise.

The relationship among the major frameworks can be conceptualized as follows.

AMTEF asks the broader question:

How should the technology be evaluated across technical, functional, economic, human, social, ethical, environmental, governance, risk, resilience, and lifecycle dimensions?

NIST AI RMF asks:

How should AI-related risks be governed, mapped, measured, and managed?

ISO/IEC 42001 asks:

How can an organization establish and operate a systematic management system for AI?

ISO/IEC 23894 asks:

How can AI-specific risk management be integrated into organizational AI activities?

OECD principles provide a broader policy-oriented framework concerning trustworthy and human-centred AI.

UNESCO provides a broader ethical and human-rights-oriented framework concerning AI and human dignity, fairness, inclusion, sustainability, and human oversight.

INTEGRATED AI EVALUATION CONCEPTUAL FRAMEWORK

Integration of AMTEF, AI-Specific Risk Evaluation, NIST AI RMF, and Organizational Governance

These frameworks can therefore operate together rather than being viewed as alternatives. The integrated structure below positions multidimensional technology evaluation as the foundation, followed by AI-specific risk identification and mapping, NIST Govern–Map–Measure–Manage activities, organizational AI management, controls and accountability, continuous monitoring, and learning and reassessment.

Figure: Integrated Evaluation Structure for Responsible AI

1. AMTEF MULTIDIMENSIONAL EVALUATION Technical • Economic • Organizational • Human/Social • Environmental
2. AI-SPECIFIC RISK IDENTIFICATION AND MAPPING Identify AI-specific risks, impacts, stakeholders, contexts, and risk relationships
3. NIST GOVERN – MAP – MEASURE – MANAGE Governance → Context & Risk Mapping → Measurement → Risk Management
4. ORGANIZATIONAL AI MANAGEMENT Policies • Roles • Processes • Resources • Governance structures
5. CONTROLS AND ACCOUNTABILITY Controls • Responsibilities • Documentation • Oversight • Transparency
6. CONTINUOUS MONITORING Performance • Risk indicators • Compliance • Emerging impacts
7. LEARNING AND REASSESSMENT Feedback • Lessons learned • Model/system reassessment • Improvement

Interpretation of the framework. The model is conceived as an integrated and iterative evaluation structure rather than a sequence of mutually exclusive frameworks. AMTEF establishes the multidimensional evaluation foundation; AI-specific risk identification and mapping makes the assessment sensitive to the distinctive risks of AI systems; the NIST Govern–Map–Measure–Manage functions provide an operational risk-management cycle; organizational AI management translates evaluation into governance and implementation; controls and accountability establish responsibility and oversight; continuous monitoring provides evidence of ongoing performance and risk; and learning and reassessment closes the loop by feeding new evidence back into subsequent evaluation and management.

Source: Author's integrated conceptualisation.

This integrated perspective is particularly important because AI risks rarely remain within one category. A cybersecurity incident can become a privacy incident. A data-quality problem can become a fairness problem. A model error can become a financial problem. A biased decision can become a reputational or legal problem. A generative-AI hallucination can become an operational or safety problem. A large-scale AI infrastructure requirement can create environmental and resource consequences.

AI risk management should therefore recognize risk interactions rather than evaluating each risk in complete isolation.

The evaluator should also distinguish inherent risk from residual risk. Inherent risk represents the risk associated with the technology before controls are considered. Residual risk represents the remaining risk after controls have been implemented. A system may have substantial inherent risk but relatively low residual risk if effective controls exist. Conversely, a seemingly moderate inherent risk may become significant if controls are weak or poorly implemented.

Risk controls should also be evaluated for effectiveness rather than simply documented. An organization may have an AI policy, incident-response procedure, model-validation process, or human-approval requirement on paper while those mechanisms are ineffective in practice.

The appropriate evaluation question is therefore not merely whether AI risks have been identified, but whether the organization can demonstrate that significant risks have been understood, measured, controlled, monitored, communicated, and reassessed over time.

10.23 Evaluating Generative AI

Generative AI requires evaluation of:

factual reliability;

hallucination;

intellectual property;

privacy;

bias;

harmful content;

cybersecurity;

transparency;

human oversight;

energy/resource implications;

workforce impact;

educational impact; and

organizational governance.

The evaluation should distinguish between low-risk productivity applications and high-impact applications.

Generative AI differs from many conventional AI systems because its outputs are not limited to a predefined classification or numerical prediction. It can generate text, images, audio, video, software code, summaries, recommendations, explanations, translations, synthetic data, and other forms of content. This broad generative capability creates a correspondingly broad evaluation problem.

The first question should concern purpose. The evaluator should establish what the generative system is intended to accomplish and whether generative AI is actually necessary for that purpose. A conventional database, search system, rules-based application, deterministic software process, or human workflow may sometimes achieve the same objective with different risk characteristics.

Factual reliability is particularly important. Generative systems can produce information that is linguistically persuasive while being factually incorrect. Evaluation should therefore include representative factual tests, domain-specific questions, source verification, calculation checks, citation verification, and tests involving uncertainty.

Hallucination should be evaluated according to the intended use. The acceptable error rate for a creative writing assistant may differ substantially from the acceptable error rate for an academic research assistant, medical support system, legal information system, financial application, or safety-critical system.

Intellectual property should also be considered. Generative AI may be trained, fine-tuned, retrieved, or otherwise developed using large collections of information. Evaluation should therefore consider the organization's responsibilities concerning copyrighted materials, proprietary information, confidential information, generated content, licensing arrangements, attribution, and downstream use.

Privacy is another major concern. Generative systems can process large volumes of user-provided information, including documents, communications, customer information, organizational records, and potentially sensitive personal data. The evaluator should determine what information is transmitted to the system, where it is processed, how it is stored, whether it is used for subsequent model development, who can access it, and how long it is retained.

Bias should be assessed at several stages. Training data may contain historical or representational biases. The model may reproduce stereotypical associations. Prompt design may influence outputs. Users may selectively use outputs. Organizational workflows may amplify certain patterns. Evaluation should therefore consider both generated content and the processes surrounding its use.

Harmful content evaluation should consider the possibility that a system may generate unsafe, discriminatory, abusive, misleading, or otherwise inappropriate material. The exact evaluation criteria should reflect the intended application, user population, deployment environment, and consequence profile.

Cybersecurity requires particular attention because generative AI systems often have interactive interfaces and may be connected to external tools, databases, retrieval systems, or organizational applications. Prompt injection, malicious instructions, data leakage, unauthorized tool use, and compromised dependencies may therefore create risks beyond those associated with conventional text-generation systems.

Transparency should include information about system capabilities and limitations. Users should understand whether they are interacting with a generative system, what types of outputs it can produce, what information sources are available, and where verification is required.

Human oversight should be matched to the consequences of use. A system that generates preliminary brainstorming material may require relatively limited oversight. A system generating content that directly affects customers, employees, patients, students, financial transactions, legal decisions, or public communications may require substantially stronger review.

Energy and resource implications should also be considered. Generative AI can require substantial computational resources during model development and operation. Evaluation should therefore consider the broader environmental lifecycle, including computing infrastructure, energy consumption, cooling, hardware requirements, data storage, network activity, model updates, and eventual hardware replacement.

Workforce impact should be evaluated beyond the simplistic question of whether AI creates or eliminates jobs. AI can alter task composition, required skills, supervision structures, workload, professional autonomy, productivity expectations, and responsibility. Some roles may experience task substitution while others experience task augmentation or transformation.

Educational impact requires similar multidimensional consideration. Generative AI can support tutoring, language learning, accessibility, feedback, content development, research assistance, and personalized learning. At the same time, inappropriate use may affect assessment validity, academic integrity, student independence, critical thinking, and the development of foundational skills. Evaluation should therefore consider both educational opportunities and educational risks.

Organizational governance should establish approved uses, prohibited uses, data-handling requirements, verification requirements, accountability, incident procedures, procurement standards, employee training, and review mechanisms.

Generative AI should also be evaluated for reproducibility and consistency. Because generative systems may produce different outputs for similar prompts, organizations should determine whether variability is acceptable for the intended purpose.

Prompt dependence is another consideration. The quality of an output may depend heavily on how instructions are formulated. Evaluation should therefore test whether ordinary users can obtain reliable results or whether safe and effective use requires specialized expertise.

Context-window limitations, retrieval errors, outdated information, tool failures, and conflicting source material should also be considered. A generative system may provide an accurate answer when sufficient context is available but produce an unreliable answer when relevant information is missing.

The evaluation should therefore move beyond the question:

Can the system generate good content?

The more important questions are:

Can the system generate content that is appropriate for the intended purpose?

Can users recognize when the content is unreliable?

Can the organization verify important outputs?

Can errors be detected before they cause significant consequences?

Can the system be controlled when it behaves unexpectedly?

Can affected stakeholders obtain appropriate explanation, correction, or review?

Generative AI should consequently be evaluated as a socio-technical capability rather than merely as a content-generation engine.

A useful generative-AI evaluation sequence is:

Purpose → data and information sources → model capability → output quality → verification → human interaction → organizational use → stakeholder consequences → monitoring → governance → reassessment.

The distinction between low-risk productivity applications and high-impact applications should remain central. The same underlying generative model may be relatively low-risk when used to produce preliminary meeting notes but potentially much higher risk when integrated into a system that automatically communicates with customers, makes financial commitments, provides medical information, assesses students, or influences employment decisions.

Risk classification should therefore focus not only on the underlying model but also on the application, level of autonomy, affected population, reversibility of decisions, access to external systems, and consequences of error.

10.24 Evaluating AI Agents

AI agents introduce additional complexity because they can perform sequences of actions rather than simply generate outputs.

Evaluation should therefore examine:

action authorization;

tool access;

persistence;

autonomy;

escalation;

error recovery;

logging;

human approval;

external communication;

financial transactions; and

ability to cause irreversible consequences.

The more autonomous the system, the greater the need for clearly defined boundaries.

AI agents represent an important development in the evolution of artificial intelligence because they can move from generating recommendations or content toward executing actions on behalf of users or organizations. An agent may interpret a goal, develop a sequence of steps, access tools, retrieve information, communicate with other systems, make decisions, and continue operating across multiple stages.

This changes the evaluation problem fundamentally.

A conventional AI model may produce an incorrect answer. An AI agent may produce an incorrect answer and then act upon it.

The consequences of this distinction can be substantial. If an agent has access to organizational databases, email systems, financial platforms, procurement systems, customer accounts, software-development environments, or operational infrastructure, an erroneous decision can potentially result in real-world consequences.

The first evaluation question should therefore concern authorization. The evaluator should determine exactly what actions the agent is permitted to take and whether those permissions are necessary for its intended purpose.

Least privilege becomes particularly important. An agent should not automatically receive unrestricted access to every system that could potentially be useful. Access should be limited according to the minimum capabilities required for the approved task.

Tool access should also be evaluated separately from model capability. An AI model may be relatively low-risk when it can only produce text but become substantially more consequential when connected to tools capable of sending messages, executing code, changing databases, making purchases, transferring funds, modifying records, or controlling physical systems.

Persistence introduces another dimension. A persistent agent may retain context, memory, tasks, permissions, or objectives across multiple interactions. Evaluation should therefore examine what information persists, how long it persists, who can modify it, and whether incorrect information can become embedded in future decisions.

Autonomy should be explicitly defined rather than treated as a binary characteristic. An agent may operate entirely under human instruction, operate within predefined workflows, initiate actions subject to approval, or perform extended sequences independently. Each level creates different control requirements.

Escalation mechanisms are therefore essential. The agent should know when a task exceeds its authority, when uncertainty becomes significant, when conflicting instructions are encountered, when sensitive information is involved, or when an action could create substantial consequences.

Human approval should be proportionate to the action. Some actions may be safely automated. Others may require confirmation before execution. High-consequence or irreversible actions may require explicit human authorization.

Error recovery is another important dimension. Agents can make mistakes at one stage that subsequently influence later stages. Evaluation should therefore examine whether the agent can detect errors, stop execution, reverse actions where possible, seek clarification, and escalate to a human.

Logging becomes particularly important for agents because a single final output may not adequately describe how the system reached an outcome. An audit trail may need to capture the initial instruction, information retrieved, tools accessed, intermediate decisions, actions attempted, actions completed, errors encountered, human approvals, and final outcome.

External communication creates additional risk. An agent capable of sending emails, messages, reports, customer communications, or public content may unintentionally communicate incorrect or unauthorized information. Evaluation should therefore examine message authorization, recipient verification, content verification, confidentiality, and escalation requirements.

Financial transactions represent another high-consequence category. An agent capable of purchasing goods, transferring funds, approving invoices, changing prices, or entering contractual commitments should operate within clearly defined financial limits and approval procedures.

The ability to cause irreversible consequences is perhaps one of the most important characteristics distinguishing agent evaluation from conventional model evaluation. An incorrect recommendation can sometimes be ignored. An incorrect deletion of records, transfer of funds, publication of confidential information, termination of an account, or modification of critical infrastructure may be difficult or impossible to reverse.

The evaluator should therefore classify agent actions according to reversibility.

Reversible actions may be automatically executed under defined conditions.

Partially reversible actions may require additional safeguards.

Irreversible or high-consequence actions should generally require stronger authorization, verification, and audit mechanisms.

Agent evaluation should also consider goal misinterpretation. An agent may follow the literal wording of an objective while failing to understand the broader organizational intention. Consequently, evaluation should include ambiguous instructions, conflicting objectives, incomplete information, unexpected environmental conditions, and situations where the most obvious action is not necessarily the appropriate action.

Agents may also encounter conflicting instructions from users, tools, retrieved documents, websites, APIs, or other agents. The system therefore needs mechanisms for determining which instructions are authoritative.

Prompt injection and indirect instruction manipulation become particularly important when agents retrieve information from external sources. A malicious document or webpage could contain instructions designed to influence the agent's subsequent actions. The evaluation should therefore test whether the agent can distinguish trusted instructions from untrusted information.

Multi-agent environments create additional complexity. When several agents interact, an error or malicious instruction can propagate from one agent to another. Evaluation should therefore consider communication boundaries, authentication, authorization, message integrity, role definitions, and failure containment.

AI agents should also be evaluated for resource consumption. An autonomous agent may repeatedly call external tools, generate unnecessary computations, consume cloud resources, or create excessive transactions. Limits should therefore exist concerning time, computational resources, API calls, spending, and other operational parameters.

A useful agent-control structure can be represented as:

Goal → authorized capabilities → information access → planning → action proposal → verification → human approval where required → execution → monitoring → logging → error recovery → escalation.

The level of autonomy should be proportional to the consequence of action.

This leads to an important principle for AI-agent evaluation:

The greater the potential consequence, irreversibility, external reach, and autonomy of an AI agent, the stronger the requirements for authorization, verification, monitoring, auditability, human oversight, and recovery.

Agent evaluation should therefore move beyond conventional questions such as whether the model generates accurate answers. The evaluator should also ask whether the agent can be trusted to act within clearly defined boundaries, whether its actions can be reconstructed after the fact, whether unauthorized actions can be prevented, whether mistakes can be contained, and whether humans retain meaningful control over consequential decisions.

The evaluation of AI agents can consequently be integrated into the broader AMTEF structure by considering technical capability, functional value, economic value, human consequences, social effects, ethical implications, cybersecurity, privacy, environmental impact, governance, legal exposure, systemic risk, resilience, and lifecycle consequences.

An AI agent should therefore not be evaluated solely according to how intelligently it appears to reason or how successfully it completes tasks. The more fundamental evaluation question is whether the agent can perform authorized actions reliably, safely, transparently, accountably, and within boundaries that remain appropriate as conditions change.

The complete agent lifecycle can be conceptualized as:

Purpose definition → capability specification → permission design → tool integration → testing → controlled deployment → action monitoring → human oversight → incident management → performance review → permission modification → continued operation or retirement.

This places AI-agent evaluation within the broader principle established throughout this chapter: the value and risk of artificial intelligence cannot be understood from model performance alone. AI must be evaluated according to what it does, what it can do, who controls it, what information it can access, what actions it can take, who may be affected, what happens when it fails, and whether the organization can retain meaningful control throughout its lifecycle.

10.25 AI Evaluation Matrix

10.26 Case Study: AI Recruitment System

Suppose an organization introduces an AI recruitment system.

The system ranks applicants according to predicted job suitability.

Technical evaluation might show high predictive performance.

However, a complete AMTEF evaluation would also examine:

historical recruitment bias;

gender differences;

socioeconomic effects;

disability access;

explainability;

applicant privacy;

legal requirements;

human review;

appeal mechanisms;

vendor dependency.

Obermeyer et al. (2019) demonstrated how apparently efficient healthcare algorithms can produce racial disparities when the target variable does not adequately represent the underlying need. The broader lesson is that model optimization does not automatically guarantee equitable outcomes.

The recruitment example illustrates why artificial intelligence cannot be evaluated solely according to predictive accuracy. An organization may develop or procure a system that appears technically effective because it predicts which applicants resemble historically successful employees. However, historical employment outcomes may themselves reflect unequal access to opportunities, differences in educational opportunities, organizational preferences, discriminatory practices, socioeconomic conditions, or other structural factors. A model trained on historical outcomes can therefore reproduce patterns from the past while presenting them as apparently objective predictions about the future.

The first evaluation question should therefore concern the purpose of the recruitment system. What exactly is the system intended to accomplish? Is it intended to identify applicants who are likely to perform well, reduce administrative workload, identify suitable candidates, improve consistency, reduce recruitment costs, or support human decision-makers? Different purposes require different evaluation criteria.

The evaluator should also ask whether AI is necessary. Recruitment organizations may use conventional application screening, structured interviews, skills assessments, human review, work samples, reference checks, or combinations of these approaches. AI should therefore be evaluated against realistic alternatives rather than being assumed to be the appropriate solution simply because it is technologically available.

The definition of job suitability is particularly important. A model cannot predict an abstract concept without operationalizing it through measurable variables. If job suitability is represented by previous performance ratings, promotion history, retention, salary progression, attendance, manager evaluations, or other historical measures, the evaluator must determine whether these variables adequately represent the actual organizational objective.

For example, an employee's historical promotion may reflect managerial preferences rather than actual capability. Retention may reflect workplace conditions rather than employee quality. Performance ratings may differ between managers. Absence records may be influenced by disability or caregiving responsibilities. Consequently, the target variable can become a major source of bias.

The data used by the recruitment system should therefore be evaluated carefully. The evaluator should examine where applicant data came from, which populations are represented, whether historical data are complete, how missing information is handled, how variables were selected, whether sensitive information is included, and whether proxy variables may indirectly reproduce protected or socially significant characteristics.

The recruitment process itself should also be examined. AI may be used for résumé screening, candidate ranking, interview scheduling, video analysis, psychometric assessment, communication, or final recommendation. Each application creates a different risk profile.

An AI system that merely assists with administrative scheduling may have relatively limited decision-making authority. A system that automatically rejects applicants before a human review occurs has substantially greater consequences because the AI becomes a gatekeeper to employment opportunities.

This distinction between assistance and decision authority is essential. The same underlying model may create different consequences depending on how it is integrated into organizational processes.

Gender differences should be evaluated using appropriate evidence. The evaluator should examine whether selection rates, false-positive rates, false-negative rates, ranking distributions, and other relevant outcomes differ systematically among groups. However, observed differences should be interpreted carefully. A statistical difference alone does not establish the cause of the difference. The evaluator should investigate whether differences arise from data representation, job requirements, measurement processes, model behaviour, workflow decisions, or other factors.

Socioeconomic effects should also be considered. Historical employment data may reflect unequal educational opportunities, differences in access to professional networks, geographical differences, language opportunities, technology access, and other socioeconomic conditions. If these characteristics influence model predictions, the system may reproduce existing inequalities.

Disability access requires particular attention. AI recruitment systems may rely on résumé formats, online assessments, automated video interviews, speech recognition, visual analysis, timed tests, or other interaction mechanisms that are not equally accessible to all applicants. The evaluator should therefore examine whether applicants can participate through appropriate alternative mechanisms and whether accessibility requirements are incorporated into system design rather than treated as an afterthought.

Explainability is also important. Applicants and recruitment personnel may need to understand why an applicant received a particular ranking or was excluded. An explanation should be sufficiently meaningful to support review and, where appropriate, correction. A generic statement such as "the algorithm determined that the applicant was not suitable" provides little meaningful accountability.

The organization should also examine whether applicants can challenge or appeal an AI-supported decision. An appeal mechanism should not merely exist formally. The applicant should know how to raise a concern, the organization should have a process for reviewing the concern, and the review should be capable of reconsidering the original AI-supported outcome.

Human review should therefore be meaningful. If recruiters simply accept the ranking generated by the system, the existence of a human reviewer may provide little additional protection. Recruiters should understand the system's limitations and have the authority and opportunity to question recommendations.

Human oversight should also be monitored. The organization could examine override rates, patterns in overrides, disagreement between recruiters and AI recommendations, and whether certain groups are disproportionately affected by automated decisions.

Privacy represents another major dimension. Recruitment systems can process résumés, employment histories, educational records, contact details, assessment results, interview recordings, behavioural information, and potentially sensitive personal information. The organization should therefore establish what information is collected, why it is required, how it is protected, who can access it, how long it is retained, and whether it is transferred to third parties.

Vendor dependency is also important. If the recruitment system is supplied by an external company, the organization may have limited visibility into model architecture, training data, update schedules, performance changes, or subcontractors. The organization should therefore evaluate contractual arrangements, audit rights, service-level commitments, data-processing responsibilities, security requirements, incident notification, model updates, and system exit arrangements.

A vendor may update the model without the organization's direct involvement. Such an update could alter performance or fairness characteristics. Consequently, the organization should require appropriate notification and reassessment mechanisms for material changes.

Legal and regulatory requirements should also be mapped according to the jurisdiction in which recruitment takes place. Requirements may concern discrimination, privacy, employment practices, accessibility, automated decision-making, record keeping, and data protection. The evaluator should not assume that a technically successful system is automatically legally or organizationally acceptable.

The economic evaluation should examine both direct and indirect costs. Direct costs may include software licensing, implementation, integration, training, maintenance, cloud services, data storage, and vendor fees. Indirect costs may include litigation exposure, reputational damage, employee relations problems, applicant dissatisfaction, remediation, and the cost of correcting inappropriate decisions.

The system should also be evaluated for operational effectiveness. Does it actually reduce recruitment time? Does it improve candidate identification? Does it reduce administrative workload? Does it improve consistency? Does it create new work for recruiters because every AI recommendation requires verification? A technology that appears efficient at the model level may create additional organizational costs elsewhere.

The social consequences should also be considered. Recruitment is not merely an internal organizational process. Employment decisions affect income, career development, family welfare, social mobility, professional identity, and access to economic opportunities.

Environmental considerations may also become relevant when recruitment systems operate at very large scale or rely on computationally intensive AI services. Although the environmental consequences of a recruitment application may be smaller than those associated with large-scale industrial AI systems, a comprehensive AMTEF evaluation should not assume that digital systems have no environmental footprint.

The evaluation can therefore proceed across several connected layers.

At the technical level, the evaluator examines predictive performance, calibration, robustness, data quality, and model stability.

At the functional level, the evaluator examines whether the system actually improves recruitment activities.

At the economic level, the evaluator examines costs, benefits, productivity, and long-term sustainability.

At the human level, the evaluator examines applicants, recruiters, accessibility, understanding, and human oversight.

At the social level, the evaluator examines distributional effects, equal opportunity, and broader labour-market consequences.

At the ethical level, the evaluator examines fairness, dignity, autonomy, transparency, and appropriate use.

At the legal level, the evaluator examines applicable requirements and responsibilities.

At the governance level, the evaluator examines ownership, accountability, monitoring, auditing, and escalation.

At the risk level, the evaluator examines failure modes, uncertainty, cybersecurity, privacy, vendor dependency, and potential cascading consequences.

The case therefore demonstrates the central AMTEF principle that a high-performing technology is not automatically a high-value technology. Technical effectiveness is one component of evaluation rather than the final conclusion.

A simplified evaluation sequence for the AI recruitment system can be represented as:

Recruitment objective → job-suitability definition → alternative approaches → data evaluation → model development → technical validation → subgroup evaluation → accessibility assessment → human review → privacy assessment → legal assessment → deployment → monitoring → applicant feedback → incident management → reassessment.

The system should also be evaluated longitudinally. A model that performs acceptably during initial implementation may behave differently as labour-market conditions change, job requirements evolve, organizational priorities change, or the composition of applicant’s changes.

For this reason, recruitment AI should not receive permanent approval based solely on its original validation results. Approval should be accompanied by monitoring requirements and conditions for reassessment.

The case ultimately demonstrates an important distinction between prediction and decision-making. An AI system may predict that one applicant resembles previous successful employees more closely than another applicant. That prediction does not, by itself, establish that the first applicant should be hired. Human judgment, organizational requirements, fairness considerations, legal requirements, contextual evidence, and other relevant information may still be necessary.

The AI system should therefore be treated as one component within the recruitment decision architecture rather than as the recruitment decision itself.