Part III — Evaluating Risk, Intelligent Technologies, Digital Systems, Biotechnology and Human-Centred Technologies

Chapter 10. Evaluating Artificial Intelligence and Intelligent Technologies — Part 1 of 3

10.1 Introduction

Artificial intelligence represents one of the most significant areas of contemporary technological development. AI systems increasingly support decision-making, prediction, classification, content generation, optimization, diagnosis, education, customer service, scientific research and organizational processes.

The significance of AI does not arise solely from computational performance. AI can influence who receives opportunities, services, resources and information. It can also alter employment, organizational structures and professional responsibilities.

Consequently, AI evaluation must extend beyond accuracy.

The NIST AI Risk Management Framework identifies trustworthy AI considerations that include characteristics such as validity and reliability, safety, security and resilience, accountability and transparency, explainability and interpretability, privacy enhancement and fairness with harmful bias managed (National Institute of Standards and Technology [NIST], 2023). NIST subsequently published a Generative AI Profile to address risks specific to generative systems (NIST, 2024a).

10.2 What Is Being Evaluated?

AI evaluation can occur at several levels:

the data;

the algorithm;

the model;

the application;

the human-AI interaction;

the organization;

the social environment; and

the wider ecosystem.

A model can therefore be technically strong while an application built around it remains inappropriate. This distinction is fundamental because artificial intelligence should not be evaluated solely according to the mathematical or computational performance of the underlying model. An AI system is generally part of a larger socio-technical arrangement involving data, software, hardware, users, organizational processes, institutional rules, external suppliers, and social expectations. Consequently, the quality of one component does not automatically establish the quality, appropriateness, or safety of the overall system.

At the data level, evaluation considers where the data originated, how they were collected, whether they are sufficiently accurate and representative, how they were labelled, whether they contain missing or inconsistent information, and whether their use is appropriate for the intended purpose. Data can introduce limitations into an AI system before any algorithm or model is applied. If the underlying data do not adequately represent the population or operating environment, a highly sophisticated model may reproduce or amplify those limitations.

At the algorithm level, evaluation considers the computational procedure used to transform inputs into outputs. Different algorithms may make different assumptions about relationships among variables, patterns in data, optimization objectives, and acceptable error. Algorithm selection should therefore be related to the problem being addressed rather than determined simply by technical popularity or complexity.

At the model level, evaluation examines the trained system that results from applying an algorithm to data. Relevant considerations may include predictive performance, generalization, robustness, calibration, sensitivity to changing conditions, interpretability, computational requirements, and stability. A model should be assessed under conditions that are relevant to its intended use rather than only under ideal development conditions.

At the application level, the question changes from whether the model works to whether the complete application works appropriately. The application may contain interfaces, business rules, databases, APIs, human review procedures, automated workflows, and decision thresholds. A technically accurate model can therefore be incorporated into an application that generates inappropriate outcomes because of poor integration, incorrect configuration, inappropriate thresholds, or unsuitable use.

At the human-AI interaction level, evaluation examines how people understand, use, question, accept, reject, or override AI outputs. Human users may over-rely on automated recommendations, misunderstand confidence levels, ignore warnings, or reject useful recommendations because the system is difficult to understand. The interaction between human judgment and AI output can therefore become an important source of both value and risk.

At the organizational level, evaluation considers whether the organization has the necessary governance, expertise, resources, policies, controls, accountability structures, and monitoring mechanisms to use AI responsibly. An organization may acquire a technically capable AI system but lack the personnel or governance processes necessary to operate it effectively.

At the social-environment level, evaluation considers how AI affects individuals, communities, institutions, labour markets, access to services, social relationships, and patterns of participation. The consequences of AI may extend beyond the organization that deploys it.

At the wider ecosystem level, evaluation considers relationships among developers, vendors, cloud providers, data providers, regulators, users, infrastructure providers, and other interconnected actors. This is increasingly important because AI systems frequently depend upon external models, datasets, application programming interfaces, cloud infrastructure, software libraries, and specialized hardware.

The evaluation can therefore be represented as:

Data → algorithm → model → application → human interaction → organization → society → ecosystem.

The important principle is that AI quality is multidimensional. A high-performing model does not automatically constitute a high-quality AI system. Evaluation should consider whether the technology is technically effective, functionally appropriate, secure, explainable where necessary, fair in the relevant context, environmentally and economically sustainable, legally permissible, organizationally governable, and appropriate for the people and systems affected by it.

10.3 THE AI LIFECYCLE

Conceptual Framework

Figure 10.3: The AI Lifecycle

The lifecycle is iterative: monitoring may lead to further data preparation, retraining, updating, redeployment, or retirement.

Suresh and Guttag (2021) demonstrate why harms can arise at multiple stages of the machine-learning lifecycle rather than solely at the algorithmic stage.

The lifecycle perspective is important because AI systems do not remain static. Their data may change, their operating environments may change, users may change, models may be retrained, software may be updated, and organizational purposes may evolve. An evaluation performed before deployment may therefore become incomplete as the system changes.

Problem definition establishes the purpose of the AI system. If the problem itself is poorly defined, technical optimization may simply produce an efficient solution to the wrong problem. The organization should therefore establish what decision, process, prediction, classification, generation, or operational function the AI system is intended to support.

Data collection determines the information available to the system. Questions concerning data source, consent or lawful basis where relevant, sampling, representativeness, data ownership, collection methods, and potential measurement error become important at this stage.

Data preparation can involve cleaning, transformation, labelling, normalization, feature engineering, filtering, integration, and other processes. These activities may alter the information available to the model and can therefore influence its subsequent performance and limitations.

Model development involves selecting an appropriate modelling approach and determining how the system will translate information into outputs. Model development should consider the intended use, available evidence, computational requirements, explainability requirements, security considerations, and consequences of error.

Training is the stage in which the model learns patterns or relationships from training data. Training performance alone should not be interpreted as evidence that the model will perform effectively in operational conditions.

Validation is used to support model selection and tuning. It provides information that can help developers compare alternative configurations and identify potential overfitting.

Testing provides an additional evaluation of performance using data or conditions intended to provide evidence about generalization. Test data should be managed carefully so that the evaluation remains meaningful.

Deployment introduces the model into its actual or intended operational environment. This may expose the system to conditions that were not present during development, including unexpected user behaviour, changing data distributions, infrastructure constraints, adversarial behaviour, or integration problems.

Monitoring becomes essential after deployment. Monitoring can examine performance, error patterns, data drift, model drift, system availability, security events, user behaviour, and other relevant indicators.

Updating may involve retraining, recalibration, software modification, changes in data sources, changes in configuration, or replacement of the model. Every material update can potentially alter system behaviour and should therefore trigger appropriate evaluation.

Retirement is also part of the AI lifecycle. A model may become obsolete, unsupported, inaccurate, insecure, uneconomical, or inappropriate because the original problem has changed. Retirement should include appropriate data handling, access removal, documentation, transition planning, and secure disposal where applicable.

The lifecycle can therefore be viewed as a continuous rather than linear process:

Develop → deploy → monitor → learn → update → redeploy → reassess → retire.

This reinforces the principle that AI evaluation should continue throughout the operational life of the system.

10.4 Problem Definition

The first AI evaluation question should be:

Is AI actually an appropriate solution to the problem?

Organizations sometimes begin with a technology and then search for applications.

A better approach begins with a defined problem.

Evaluators should ask:

What problem is being solved?

Who experiences the problem?

What alternatives exist?

Why is AI necessary?

What would happen if AI were not used?

What are the consequences of error?

Problem definition is one of the most important stages because an inappropriate problem definition can cause subsequent technical work to become misdirected. An organization may successfully build a technically sophisticated AI system without demonstrating that AI is the most appropriate method for addressing the underlying need.

The problem should first be expressed independently of the technology. For example, “we need to improve the allocation of medical appointments” describes a problem, whereas “we need an AI scheduling model” describes a proposed technological solution. Separating the problem from the proposed solution allows alternative approaches to be considered.

Alternative solutions may include conventional statistical methods, rules-based systems, process redesign, additional human staffing, improved information systems, organizational changes, or no technological intervention at all. AI should be justified in relation to the problem rather than adopted simply because it is technologically available.

The consequences of error should be identified at this stage. An incorrect recommendation in a low-risk commercial application may create inconvenience or financial inefficiency. An incorrect recommendation in healthcare, financial services, emergency response, education, employment, or public services may have substantially greater consequences.

Problem definition should also consider who benefits and who may be affected. A system designed to improve organizational efficiency may alter employee responsibilities. A system designed to reduce costs may change access to services. A system designed to automate decisions may affect opportunities or obligations for individuals.

The evaluation should also consider whether AI is being used for a task that requires prediction, classification, pattern recognition, generation, optimization, or another capability for which AI provides meaningful value. The use of AI should have a defensible relationship with the problem.

Another important question is whether the organization has defined success appropriately. Success should not necessarily be defined only in terms of model accuracy. It may include service quality, reduced processing time, improved access, reduced errors, cost effectiveness, user satisfaction, safety, fairness, or other outcomes.

The problem-definition stage should therefore establish:

Problem → affected stakeholders → desired outcome → alternatives → role of AI → acceptable error → evaluation criteria.

This provides the foundation for all subsequent stages.

10.5 Data Quality

AI systems depend heavily upon data.

Data quality dimensions include:

accuracy;

completeness;

consistency;

relevance;

timeliness;

representativeness;

provenance; and

integrity.

Poor data can produce unreliable models even when sophisticated algorithms are used.

Data quality should therefore be evaluated as a substantive component of AI performance rather than as a preliminary technical housekeeping activity. The quality of the model can be constrained by the quality of the information available to it.

Accuracy concerns whether data correctly represent the characteristics they are intended to describe. Incorrect labels, measurement errors, transcription errors, duplicate records, and faulty sensors can reduce the usefulness of training and operational data.

Completeness concerns whether important information is missing. Missing data can be systematic rather than random. If particular groups or circumstances are more likely to be missing from a dataset, the resulting model may perform differently across populations.

Consistency concerns whether data are recorded and represented in compatible ways. Different departments may use different definitions, coding systems, units, or time periods. Inconsistent information can create problems when datasets are combined.

Relevance concerns whether the data actually relate to the problem being addressed. Large datasets are not automatically valuable. Information that is abundant but weakly connected to the intended decision may increase complexity without improving performance.

Timeliness concerns whether the data remain appropriate for the current operating environment. Historical data may become less representative when behaviour, markets, populations, technologies, or environmental conditions change.

Representativeness concerns whether the data adequately reflect the population, operating environment, or circumstances to which the model will be applied. A model developed from a narrow population may not generalize effectively to a broader population.

Provenance concerns the origin and history of data. Evaluators should understand where data came from, how they were collected, what transformations occurred, and whether important assumptions were introduced during processing.

Integrity concerns whether data have remained accurate and protected from inappropriate alteration. Data may be modified intentionally or unintentionally. Integrity controls are therefore important throughout the data lifecycle.

Data quality also has a temporal dimension. A dataset may have been accurate when collected but become less relevant as conditions change. AI systems operating continuously should therefore have mechanisms for identifying important changes in data distributions.

AI evaluation should also distinguish between data quality and data quantity. A very large dataset may still be inadequate if it contains systematic errors or lacks representation of important cases. Conversely, a smaller but carefully curated dataset may be more useful for certain applications.

Data quality should therefore be assessed in relation to the intended purpose, population, context, and decision consequences.

10.6 Training, Validation and Test Data

Data should be appropriately separated for model development and evaluation.

A model that performs extremely well on training data but poorly on unseen data may be overfitting.

Evaluation should therefore examine generalization rather than merely training performance.

Training data are used to develop the model. Validation data can support model selection, parameter tuning, and comparison of alternative approaches. Test data are intended to provide an independent basis for evaluating how the selected model performs on previously unseen information.

The separation between these datasets is important because repeated exposure to the same information can lead to overly optimistic performance estimates. If test information influences model development, the test set may no longer provide an independent evaluation.

Overfitting occurs when a model learns patterns that are highly specific to the training data rather than patterns that generalize effectively to new observations. A model may therefore demonstrate excellent training performance while producing substantially weaker results in operational conditions.

Underfitting represents the opposite problem, in which the model may be insufficiently capable of representing relevant patterns in the data. Evaluation should therefore consider both the ability to learn meaningful relationships and the ability to generalize.

Data leakage is another important concern. Leakage occurs when information that would not legitimately be available at the time of prediction enters the training or evaluation process. This can produce misleadingly strong results because the model has effectively received information about the outcome it is supposed to predict.

The division of data should also reflect the actual deployment environment. Random splitting may be inappropriate for some time-dependent or geographically structured problems. If the system will predict future events, temporal separation may provide a more realistic test. If the system will operate across geographic populations, geographical separation may be useful for examining generalization.

Cross-validation can provide additional evidence when appropriate. However, the choice of validation strategy should reflect the structure of the problem rather than being applied mechanically.

Evaluation should also examine subgroup performance. A model may demonstrate acceptable aggregate performance while performing substantially differently for specific groups, regions, categories, or operating conditions. Therefore, aggregate performance should not automatically be treated as sufficient evidence of generalization.

10.7 Accuracy

Accuracy is the proportion of correct predictions among total predictions.

Accuracy is useful but can be misleading in imbalanced datasets.

For example, if only 1% of cases represent a rare event, a model that predicts "no event" every time could achieve 99% accuracy while providing no useful detection.

Consequently, evaluators should select performance measures according to the decision context.

Accuracy can be expressed as:

Accuracy = Correct Predictions / Total Predictions

Although simple to interpret, accuracy assumes that correct and incorrect predictions can be treated adequately through the same overall measure. In many real-world applications, this assumption does not hold.

A false positive occurs when a model predicts an event that is not actually present. A false negative occurs when the model fails to identify an event that is actually present. The consequences of these errors can differ substantially.

For example, in fraud detection, excessive false positives may cause legitimate transactions to be blocked. In disease screening, false negatives may delay treatment. In industrial safety, failure to detect a hazardous condition may have serious consequences. In customer recommendation systems, errors may have relatively modest consequences.

Accuracy should therefore be interpreted alongside other measures and the practical consequences associated with different types of error.

Accuracy may also vary across subgroups. A model can have acceptable overall accuracy while performing differently across demographic, geographic, institutional, or operational groups. Evaluation should therefore consider whether aggregate performance conceals meaningful differences.

Accuracy should also be evaluated under realistic operating conditions. A model tested on clean laboratory data may perform differently when confronted with incomplete, noisy, delayed, or changing information.

10.8 Precision

Precision measures the proportion of positive predictions that are correct.

Precision is particularly relevant when false positives create significant costs.

Precision can be represented as:

Precision = True Positives / (True Positives + False Positives)

A high precision value indicates that when the system identifies a positive case, that prediction is frequently correct. This can be important where acting on a false positive is costly, disruptive, or harmful.

For example, an organization using AI to identify potentially fraudulent transactions may need to avoid incorrectly blocking legitimate customers. A security system generating excessive false alerts may overwhelm analysts. An employment screening system may create unnecessary exclusions if positive classifications are not sufficiently reliable.

Precision should not, however, be interpreted independently from recall. A model can achieve high precision by making relatively few positive predictions, potentially missing many genuine positive cases.

The appropriate balance therefore depends upon the purpose of the technology and the consequences associated with different errors.

Precision should also be evaluated under different operating thresholds. Many AI systems produce scores or probabilities that are subsequently converted into decisions through a threshold. Changing the threshold can alter the relationship between precision and recall. The selected threshold should therefore be justified according to the practical context.

10.9 Recall

Recall measures the proportion of actual positive cases that are correctly identified.

Recall can be represented as:

Recall = True Positives / (True Positives + False Negatives)

Recall may be particularly important when false negatives create serious consequences.

In medical screening, for example, missing a genuine disease case may have greater consequences than producing an additional false alarm.

High recall indicates that the system identifies a large proportion of the cases that genuinely belong to the positive category. However, increasing recall may sometimes reduce precision because the system identifies more cases as positive, including some that are not actually positive.

The appropriate level of recall should therefore depend upon the consequences of missed cases. In emergency detection, safety monitoring, disease screening, cybersecurity threat detection, or other high-consequence applications, evaluators may place substantial attention on false negatives.

However, recall alone is not sufficient. A system could achieve very high recall by classifying nearly every case as positive. Such a system may create excessive false positives and therefore have limited practical usefulness.

Consequently, recall should normally be interpreted alongside precision, the confusion matrix, the underlying prevalence of the event, and the operational consequences of different errors.

10.10 F-Measure

The F-measure provides a combined measure of precision and recall.

The choice of performance metric should therefore be justified in relation to the technology's intended purpose rather than selected simply because it produces a favourable result.

The commonly used F1 measure is the harmonic mean of precision and recall:

F1 = 2 × (Precision × Recall) / (Precision + Recall)

The harmonic mean makes the F1 measure sensitive to low performance in either precision or recall. A model cannot obtain a high F1 score simply by performing very well on one measure while performing poorly on the other.

The F-measure can therefore be useful when both false positives and false negatives are important. However, it should not automatically be regarded as the best performance measure for every AI system. Its usefulness depends upon the decision context.

Different applications may require different metrics. Regression systems may require measures such as mean absolute error or root mean square error. Ranking systems may require ranking-specific measures. Probabilistic systems may require calibration measures. Image-generation or language-generation systems may require task-specific evaluation combined with human assessment. Safety-critical applications may require measures related to failure probability, detection latency, or unacceptable-event rates.

The selection of performance measures should therefore begin with the intended function of the AI system and the consequences of error.

Evaluators should also consider whether performance metrics are understandable to decision-makers and stakeholders. A technically sophisticated metric may be appropriate for model developers but insufficient for governance decisions unless its meaning is clearly explained.

Performance metrics should also be monitored after deployment. Model performance can change when the operating environment changes, when user behaviour changes, when data distributions shift, or when the technology is integrated with new systems.

10.11 Fairness

Fairness in AI is not a single mathematical concept.

Different fairness criteria can produce different conclusions.

Binns (2018) demonstrates that algorithmic fairness involves competing conceptions of what constitutes a fair outcome.

Therefore, an evaluation should identify:

which groups are affected;

which fairness definition is being used;

what trade-offs arise;

whether unequal performance is justified;

whether the data reflect historical inequalities; and

who decides what constitutes an acceptable outcome.

Fairness is one of the most complex dimensions of AI evaluation because there is no universally applicable mathematical definition that resolves every social and ethical question. Different fairness approaches may emphasize equal treatment, equal outcomes, equal error rates, equal opportunities, proportionality, individual fairness, or other principles. These approaches can sometimes produce different results when applied to the same system.

The first step in evaluating fairness is therefore to identify the relevant population groups and the context in which the technology operates. Groups may differ according to demographic, geographic, socioeconomic, linguistic, institutional, or other characteristics relevant to the particular application. The relevant groups should be established according to the actual context rather than through assumptions about which characteristics matter.

The evaluation should then identify what constitutes the decision or outcome being assessed. Fairness may concern access to an opportunity, allocation of resources, classification, prediction, service quality, recommendation, pricing, risk assessment, or another outcome. A fairness assessment cannot be meaningfully conducted without first specifying what outcome is being compared.

Data can reproduce historical inequalities. If past decisions reflect unequal access, institutional discrimination, measurement differences, or other structural conditions, a model trained on historical data may learn patterns associated with those conditions. Removing an explicit demographic variable does not necessarily eliminate this problem because other variables may act as proxies for characteristics associated with group membership.

Representativeness is therefore important. If some groups are insufficiently represented in training or evaluation data, the resulting model may perform differently for those groups. Evaluation should examine not only overall performance but also relevant subgroup performance.

Fairness should also be considered across the AI lifecycle. It may be affected by problem definition, data collection, labelling, feature selection, model development, threshold selection, deployment, human interaction, and monitoring. Consequently, fairness should not be treated solely as a property of the final algorithm.

Trade-offs should be made explicit. Improving one fairness measure may sometimes change another performance measure or fairness criterion. These trade-offs should not be hidden behind a single numerical score. Decision-makers should understand what is being optimized, what is being sacrificed, and why.

The question of whether unequal performance is justified also requires careful consideration. Some differences may result from genuine differences in the relevant underlying circumstances, while others may reflect inappropriate data, flawed assumptions, or structural disadvantage. The evaluation should therefore investigate the reasons for observed differences rather than automatically treating all differences as either acceptable or unacceptable.

Human judgment is particularly important because technical metrics cannot independently determine what society or an organization should regard as an acceptable allocation of opportunities, risks, or resources. Such decisions may involve legal requirements, ethical principles, institutional values, professional obligations, and stakeholder expectations.

Fairness should also be considered in relation to access. An AI service may technically treat users similarly while still creating unequal access because some populations lack connectivity, digital literacy, language support, financial resources, or appropriate devices. Therefore, fairness evaluation should sometimes extend beyond model outputs to the wider socio-technical environment.

Human oversight is also relevant. Where an AI system produces potentially consequential classifications or recommendations, appropriate human review may provide an additional safeguard. However, human oversight does not automatically eliminate unfairness. Human decision-makers may themselves rely excessively on automated outputs or reproduce existing biases.

Fairness evaluation should therefore combine quantitative and qualitative evidence where appropriate. Quantitative analysis can identify differences in performance, error rates, or outcomes. Qualitative investigation can help explain why those differences occur and how affected stakeholders experience the system.

The fairness evaluation can be represented as:

Population → data representation → model performance → decision threshold → human interaction → outcome distribution → stakeholder consequences → monitoring.

Fairness should also be monitored after deployment. Changes in population, data, operating conditions, organizational policies, or system configuration may alter observed outcomes. A model that initially demonstrates acceptable subgroup performance may later produce different results as circumstances change.

The central principle is that fairness should be treated as a contextual, evidence-based, and continuously evaluated property of the socio-technical system rather than as a single numerical characteristic of an algorithm. A meaningful AI evaluation should make the relevant groups, fairness criteria, assumptions, trade-offs, evidence, decision authority, and monitoring arrangements explicit.

Taken together, Sections 10.2–10.11 establish an important foundation for AI evaluation. The assessment moves from defining what is actually being evaluated, through the AI lifecycle and problem definition, into data quality, model development, performance measurement, and fairness. This sequence reinforces the broader argument that AI should be evaluated as a complete socio-technical system rather than merely as a mathematical model. The subsequent dimensions of AI evaluation can therefore build upon this foundation by examining transparency, explainability, robustness, privacy, security, accountability, human oversight, environmental consequences, economic value, and wider social effects.

10.12 Bias

Bias can enter AI systems through:

data collection;

sampling;

measurement;

labelling;

feature selection;

model development;

deployment;

human interpretation.

Suresh and Guttag (2021) provide a useful lifecycle perspective for understanding these sources of harm.

The evaluation question should therefore be:

Where in the lifecycle could systematic disadvantage arise?

Bias in artificial intelligence should not be understood simply as a technical defect located inside an algorithm. It can emerge from decisions made long before a model is trained and can continue to develop after a system has been deployed. The source of bias may therefore exist in the social environment from which the data were obtained, in the way a problem was defined, in the population selected for analysis, in the variables chosen for measurement, in the labels assigned to observations, in the assumptions embedded in a model, in the threshold used to convert predictions into decisions, or in the manner in which human users interpret and act upon system outputs.

The first stage of evaluation should therefore examine the problem formulation itself. A system can produce statistically accurate predictions while addressing a problem that has been framed in a way that disadvantages particular groups. For example, if an organization defines the objective of an AI system exclusively around historical purchasing behaviour, previous disciplinary records, past loan repayment patterns, or historical healthcare utilization, the resulting system may reproduce patterns embedded in those historical records without adequately considering why those patterns existed.

Sampling is another important source of bias. If the training population differs substantially from the population in which the system will ultimately operate, the model may perform differently across groups. A facial-recognition system trained predominantly on one demographic population, for example, may exhibit different performance characteristics when applied to populations that were insufficiently represented during development. Similar problems can occur in healthcare, education, recruitment, credit assessment, language processing, and predictive maintenance when the training environment does not adequately represent the deployment environment.

Measurement bias occurs when the variables used to represent an underlying phenomenon are themselves imperfect. A variable may be convenient to measure without being an accurate representation of the concept that the organization actually intends to evaluate. Historical expenditure may not fully represent healthcare need. Previous examination performance may not fully represent learning potential. Recorded disciplinary actions may not fully represent actual behaviour. Customer complaints may reflect differences in willingness or ability to complain rather than differences in the underlying quality of service.

Labelling is another significant source of bias. Supervised learning systems depend on labels that are often produced by human beings, institutions, historical processes, or other algorithms. If the labels contain systematic errors or reflect subjective judgments, those characteristics can become embedded in the model. The question is therefore not simply whether the model predicts the labels accurately but whether the labels themselves are appropriate representations of the phenomenon being evaluated.

Feature selection can also introduce bias. Variables may appear neutral while functioning as proxies for sensitive or socially significant characteristics. Removing an explicit demographic variable does not necessarily eliminate the possibility of differential treatment because other variables may contain correlated information. Geographic location, purchasing behaviour, language, educational history, employment patterns, or network characteristics can sometimes act as indirect indicators of group membership.

Bias may also arise during model development. Choices concerning optimization objectives, thresholds, loss functions, model architecture, training procedures, missing-data treatment, and performance metrics can influence the distribution of errors across groups. A model optimized for overall accuracy may perform differently across subgroups. A system designed to minimize false positives may generate more false negatives for particular populations. Consequently, model evaluation should examine not only aggregate performance but also the distribution of errors and outcomes across relevant populations.

Deployment creates another stage at which bias can emerge. A model may perform acceptably in a controlled development environment but behave differently when integrated into a real organizational process. Users may apply recommendations inconsistently, override predictions selectively, interpret confidence scores differently, or use the system for purposes outside its original design. The surrounding workflow can therefore alter the practical consequences of the model.

Human interpretation can create further differences. Users may place excessive confidence in AI-generated recommendations, selectively accept recommendations that confirm their expectations, or interpret uncertainty as certainty. This means that fairness evaluation should include the human-AI interaction rather than treating the model as an isolated technical object.

Bias should also be examined over time. A system that initially performs similarly across groups may become less equitable as populations, behaviour, economic conditions, institutional policies, or data-generating processes change. Continuous monitoring is therefore necessary.

A comprehensive bias evaluation should consequently distinguish among bias in problem formulation, data bias, measurement bias, sampling bias, representation bias, labelling bias, algorithmic bias, deployment bias, interaction bias, and institutional or structural bias. These forms of bias can interact with one another, making it difficult to attribute an observed disparity to a single technical component.

The evaluation should also distinguish between unequal outcomes and evidence of unjustified discrimination. An observed difference between groups does not automatically establish that an AI system is responsible for the difference or that the difference is necessarily inappropriate. Conversely, apparently equal aggregate outcomes do not prove that a system is free from problematic bias. The evaluator must examine the context, decision purpose, population, variables, error distribution, institutional processes, and consequences.

The appropriate evaluation sequence can therefore be understood as:

Problem definition → population identification → data generation → sampling → measurement → labelling → feature selection → model development → threshold selection → deployment → human interaction → outcomes → monitoring → corrective action.

The central principle is that bias evaluation should be lifecycle-based rather than model-based. The evaluator should ask not simply whether an algorithm is biased, but where systematic disadvantage could enter the socio-technical system, how it can be detected, who may be affected, what consequences may arise, and what mechanisms exist to prevent, mitigate, monitor, and correct it.

10.13 Explainability

Explainability concerns the ability to understand why a system produced an output.

Not every system requires the same degree of explanation.

However, high-impact decisions may require stronger explanatory mechanisms.

Explainability is particularly important where users have rights to challenge or appeal decisions.

The concept of explainability is closely related to, but not identical with, interpretability. Interpretability generally concerns the extent to which the internal logic or structure of a system can be understood, whereas explainability may refer more specifically to mechanisms that provide understandable reasons for particular outputs. A technically complex model may therefore be accompanied by an explanation mechanism even when the underlying model itself is difficult to interpret directly.

The required degree of explainability should be proportionate to the consequences of the decision. An AI system recommending a movie may require relatively limited explanation. An AI system supporting a medical diagnosis, determining eligibility for financial services, assisting employment decisions, allocating educational opportunities, or supporting legal or regulatory decisions may require substantially stronger explanatory mechanisms.

Explainability should also be considered from the perspective of the intended recipient. A highly technical explanation may be meaningful to a data scientist but incomprehensible to an ordinary user. An explanation that satisfies a technical audit may therefore fail to satisfy the informational needs of the person affected by the decision.

The evaluator should consider whether the explanation identifies the principal factors influencing an output, whether it is consistent with system behaviour, whether it is understandable to the intended audience, whether it accurately reflects the model's reasoning process, and whether it enables an affected person or authorized reviewer to challenge an outcome.

Explanations should not create a false impression of certainty. A system may provide a plausible-sounding explanation that does not adequately capture the complexity or uncertainty of the underlying model. Consequently, explanation quality itself should be evaluated.

Explainability can also support debugging, model validation, risk management, compliance, user trust, incident investigation, and organizational learning. If a system repeatedly produces an unexpected outcome, the ability to investigate why the output occurred can be essential for determining whether the problem arose from the data, model, integration, user interaction, or deployment environment.

However, explainability also involves trade-offs. Highly detailed explanations may reveal sensitive information, proprietary information, security vulnerabilities, or personal data. An explanation system must therefore be designed with privacy and security considerations in mind.

The appropriate evaluation question is therefore not simply whether an AI system is explainable. It is whether the system provides an appropriate level and form of explanation for its purpose, users, affected stakeholders, consequences, legal environment, and risk profile.

10.14 Transparency

Transparency can concern:

data sources;

model purpose;

model limitations;

performance;

intended users;

known risks;

update procedures;

governance arrangements.

Transparency does not necessarily require publication of proprietary source code.

The objective is meaningful understanding rather than unrestricted disclosure.

Transparency is therefore broader than source-code availability. A system may be transparent in important operational respects without exposing every element of its intellectual property. Conversely, publishing source code does not necessarily make an AI system understandable to users or affected stakeholders.

A meaningful transparency framework should communicate what the system is designed to do, what it is not designed to do, what data it relies upon, what assumptions are relevant, what populations were represented during development, what performance has been observed, what limitations have been identified, and what risks remain unresolved.

Transparency should also extend to system changes. AI systems can change as models are retrained, data are updated, thresholds are modified, software dependencies are replaced, or external services are changed. A system that was evaluated at one point in time may therefore differ materially from the system currently in operation.

Documentation should consequently support version control and traceability. Model cards, dataset documentation, system documentation, risk assessments, validation records, change histories, and monitoring records can all contribute to meaningful transparency.

Transparency also has an organizational dimension. Stakeholders should understand who owns the system, who is responsible for monitoring it, who can approve changes, who responds to incidents, and who has authority to suspend or retire it.

There is also an important distinction between transparency and disclosure. Some information may need to be available to regulators, auditors, system owners, or affected individuals without being publicly released. Transparency should therefore be proportional to legitimate informational needs and balanced against privacy, cybersecurity, intellectual-property, and safety considerations.

For high-impact AI systems, transparency should enable meaningful scrutiny. The evaluator should ask whether a competent independent reviewer could obtain sufficient information to understand the system's purpose, assumptions, limitations, performance, risks, governance arrangements, and material changes.

Transparency therefore becomes an enabling condition for accountability. Without adequate information, stakeholders may be unable to determine who made a decision, why a system behaved in a particular way, whether the system was operating within its intended scope, or whether corrective action is necessary.

10.15 Hallucination and Generative AI Reliability

Generative AI systems can produce fluent but inaccurate outputs.

This creates an important evaluation distinction:

linguistic fluency is not equivalent to factual reliability.

Evaluation should therefore test:

factual accuracy;

citation accuracy;

consistency;

robustness;

domain-specific performance;

susceptibility to misleading prompts;

harmful content;

privacy leakage;

inappropriate confidence; and

human verification requirements.

NIST's Generative AI Profile specifically recognizes that generative systems create distinctive risk-management considerations (NIST, 2024a).

Generative AI introduces a particularly important challenge because the system can produce outputs that appear authoritative, coherent, detailed, and professionally written even when the underlying information is incomplete, unsupported, outdated, or incorrect. The quality of language therefore cannot be used as a substitute for the quality of information.

Hallucination should be evaluated according to the purpose and context of the system. A minor factual error in a creative-writing application may have limited consequences. A fabricated medical reference, incorrect legal interpretation, inaccurate financial calculation, false academic citation, or unsupported safety recommendation may have substantially greater consequences.

The evaluation of generative AI should therefore distinguish between different forms of reliability. Factual reliability concerns whether statements correspond with available evidence. Citation reliability concerns whether cited sources actually support the claims being made. Instructional reliability concerns whether the system follows legitimate instructions. Consistency concerns whether substantially similar prompts produce materially different answers without an appropriate reason. Robustness concerns how performance changes when prompts, wording, context, or input conditions change.

Another important issue is confidence. Generative systems may produce statements without providing a reliable representation of uncertainty. Users may therefore interpret fluent language as evidence of high confidence even when the system lacks sufficient information.

Evaluation should also test the system under deliberately challenging conditions. These may include ambiguous questions, incomplete information, conflicting information, misleading premises, adversarial prompts, long-context inputs, unusual terminology, domain-specific questions, multilingual inputs, and requests that require the system to acknowledge uncertainty.

Citation verification is particularly important in academic and professional applications. An AI system may generate a citation that appears plausible but does not exist, may attribute an argument to the wrong author, may provide an incorrect publication year, or may cite a genuine source that does not support the statement being made. Therefore, citation generation and citation verification should be treated as separate evaluation activities.

Generative AI evaluation should also consider privacy and memorization. Systems may potentially reproduce sensitive information or reveal information that users did not expect to be exposed. Evaluation should therefore consider the relationship between training data, prompts, outputs, retrieval systems, logging, and downstream storage.

Human verification requirements should be explicitly defined. A system may be appropriate for producing preliminary drafts while requiring expert verification before information is used for high-consequence decisions. The evaluation should therefore establish where human review is mandatory, recommended, or unnecessary.

The appropriate principle is that generative AI should be evaluated not merely by asking whether it produces impressive outputs, but by examining whether those outputs are accurate, appropriate, reliable, verifiable, safe, and fit for the intended purpose.