Part III — Evaluating Risk, Intelligent Technologies, Digital Systems, Biotechnology and Human-Centred Technologies

Chapter 9. Risk, Security, Governance and Regulation — Part 3 of 3

9.20 Regulatory Governance

Governance is broader than compliance.

Compliance asks:

Are we meeting the applicable requirements?

Governance asks:

Who is responsible for deciding what should be done, according to which principles, with what oversight and accountability?

This distinction is central to emerging technologies.

Compliance generally focuses on whether an organization satisfies applicable laws, regulations, standards, contractual requirements, or other formal obligations. Governance addresses the broader decision-making system through which responsibilities, principles, authority, oversight, risk management, transparency, and accountability are established.

An organization can therefore be technically compliant while still facing important governance questions. A technology may satisfy minimum legal requirements but raise questions about transparency, accountability, proportionality, stakeholder participation, long-term consequences, or organizational responsibility. Governance provides the structure through which these broader questions can be considered.

The OECD AI Principles emphasize human-centred and trustworthy AI, while UNESCO's Recommendation on the Ethics of AI places human rights, dignity, transparency, fairness, inclusion, environmental sustainability and human oversight at the centre of AI governance (OECD, 2024; UNESCO, 2021).

These principles illustrate the increasingly broad scope of technology governance. As technologies become capable of influencing decisions, allocating resources, monitoring behaviour, generating information, or performing functions previously undertaken by humans, governance must consider more than technical performance.

Regulatory governance should begin by identifying the relevant legal and regulatory environment. This may include sector-specific requirements, data protection obligations, consumer protection requirements, intellectual property rules, cybersecurity obligations, professional standards, environmental requirements, employment rules, financial regulations, medical requirements, and other applicable frameworks. The relevant requirements will depend upon the technology, jurisdiction, sector, purpose, and population affected.

Responsibility should be clearly allocated. Organizations should know who has authority to approve deployment, who monitors compliance, who owns the technology, who manages risk, who can suspend operation, and who has authority to respond to incidents. Ambiguous authority can create delays during critical situations.

Governance should also establish escalation mechanisms. Not every technology decision should be handled at the same organizational level. Routine low-risk decisions may be delegated, whereas decisions involving significant safety, privacy, financial, social, or legal consequences may require senior or independent oversight.

Human oversight is especially important for high-impact automated systems. Human oversight should not simply mean that a person is technically present somewhere in the process. The relevant question is whether the person has sufficient authority, information, competence, time, and independence to identify problems and intervene effectively.

Transparency is another governance dimension. Stakeholders may need information about how a technology operates, what data it uses, what its limitations are, who is responsible for it, and how concerns can be raised. The appropriate level of transparency depends upon the technology and context, but unexplained decision-making can make accountability difficult.

Governance should also address conflicts of interest. Individuals responsible for delivering a technology may have incentives that differ from those responsible for evaluating its risks. Independent review can therefore be valuable where consequences are significant. Independence may be achieved through separate governance functions, internal assurance, external audit, ethics review, regulatory supervision, or other mechanisms appropriate to the context.

Governance should be dynamic because regulatory environments and technologies change. New legislation may create new requirements, existing requirements may be interpreted differently, and new technology capabilities may create situations that were not anticipated when earlier regulations were developed. Organizations should therefore maintain processes for monitoring regulatory developments.

Regulatory governance also involves evidence. Organizations should be able to demonstrate how significant decisions were made, what information was considered, which controls were applied, and who approved the decision. This documentation can support accountability and later review.

An important principle is proportionality. Governance requirements should reflect the potential consequences of the technology. A low-risk internal application may require relatively limited governance, while a technology affecting safety, health, financial transactions, essential services, or significant personal data may require substantially stronger oversight.

Governance should also incorporate stakeholder perspectives where appropriate. Users, employees, affected communities, technical specialists, legal experts, regulators, and other relevant stakeholders may identify risks that are not visible from the perspective of the developer or organizational sponsor alone.

The governance sequence can therefore be understood as:

Technology purpose → applicable requirements → ethical and organizational principles → responsibility → oversight → evidence → monitoring → review → corrective action.

This broader perspective demonstrates why governance should not be reduced to a compliance checklist. Compliance is an important component of governance, but effective governance also concerns how decisions are made, who is accountable, how uncertainty is handled, and how the organization responds when circumstances change.

9.21 Regulatory Sandboxes

A regulatory sandbox allows innovative technologies to be tested under controlled conditions while regulators observe their effects.

Potential benefits include:

learning before large-scale deployment;

identifying regulatory gaps;

testing safeguards;

enabling controlled experimentation; and

improving dialogue between developers and regulators.

However, a sandbox should not be treated as an automatic declaration that a technology is safe or appropriate.

The purpose of a regulatory sandbox is generally to create a controlled environment in which innovation and regulatory learning can occur simultaneously. Emerging technologies may not fit neatly within existing regulatory categories, and regulators may need practical evidence to understand how a technology operates and what risks it creates.

A sandbox can allow regulators and technology developers to examine questions that may be difficult to resolve through theoretical analysis alone. These may include how a technology performs under real operating conditions, how users interact with it, what safeguards are effective, what information regulators need, and whether existing requirements are appropriate.

A well-designed sandbox should establish a clear scope. It should identify which technology is being tested, which functions are included, which participants are involved, what conditions apply, what data may be collected, and what period of testing is permitted.

Risk controls should be established before testing begins. The fact that a technology is innovative does not eliminate the possibility of harm during experimentation. Appropriate safeguards may include restricted user groups, limited geographic deployment, transaction limits, human oversight, monitoring requirements, emergency procedures, data protections, and predefined stopping conditions.

A sandbox should also establish evidence requirements. Developers should not simply demonstrate that a technology can operate; they may need to provide evidence concerning performance, reliability, security, user outcomes, incidents, limitations, and unexpected effects. The precise evidence required should be proportionate to the nature of the technology and its potential consequences.

Regulatory sandboxes can also help identify regulatory gaps. A technology may perform functions that existing regulations did not anticipate. Regulators can use controlled experimentation to understand whether existing requirements remain appropriate or whether additional guidance, standards, or regulatory mechanisms may be necessary.

However, sandbox participation does not automatically establish that a technology is suitable for unrestricted deployment. Results obtained under controlled conditions may not generalize to large-scale deployment. A technology may behave differently when the number of users increases, when operating conditions change, or when safeguards used in the sandbox are removed.

There is therefore a need to distinguish between sandbox evidence and general deployment evidence. The sandbox should be treated as one stage of an evidence-building process rather than the final stage of technology validation.

The results of sandbox testing should also be documented. Important findings should include observed performance, incidents, control effectiveness, unexpected behaviours, stakeholder feedback, regulatory concerns, and unresolved uncertainties. These findings can then inform subsequent technology evaluation and governance.

Regulatory sandboxes may be particularly useful in areas such as financial technology, artificial intelligence, healthcare technology, autonomous systems, digital identity, energy systems, and other areas where technological development may move faster than conventional regulatory processes.

The fundamental value of a sandbox lies in structured learning. It provides an opportunity to examine technology under defined conditions before broader implementation. Nevertheless, experimentation should remain connected to accountability, consumer or stakeholder protection, evidence generation, and appropriate regulatory oversight.

9.22 Auditability

A technology is auditable when sufficient information exists to reconstruct, examine and evaluate important decisions or processes.

Auditability may require:

system logs;

data documentation;

model versions;

decision records;

access records;

change histories;

performance monitoring; and

incident reports.

For high-impact technologies, auditability should be designed into the system rather than added after an incident.

Auditability is closely related to transparency and accountability but is not identical to either concept. Transparency concerns the availability and comprehensibility of relevant information. Accountability concerns identifiable responsibility for decisions and actions. Auditability concerns whether sufficient evidence exists to reconstruct and evaluate what occurred.

A technology may produce a technically correct result but still be difficult to audit if the organization cannot determine which version of the software was operating, which data were used, who accessed the system, what configuration was active, or what changes occurred before the decision was made.

System logs are therefore important. Logs can provide evidence of authentication, access, system changes, errors, unusual activity, and transactions. However, logs must be appropriately protected because they may contain sensitive information and may themselves become targets for manipulation.

Data documentation is similarly important. Evaluators should be able to understand the origin, transformation, classification, and use of important datasets. Without appropriate documentation, it may be difficult to reproduce an analysis or investigate whether inappropriate data contributed to an outcome.

Model versions are particularly important for AI systems. A model may change through retraining, fine-tuning, software updates, configuration changes, or changes in the underlying data. If the organization cannot identify which model version produced an important output, subsequent investigation may be difficult.

Decision records are particularly important when technology supports or influences consequential decisions. Records may need to show what information was considered, what recommendation or output was generated, whether a human reviewed the result, and what final decision was made. The level of documentation should be proportionate to the consequences of the decision.

Access records provide evidence about who interacted with a system and when. Such records can support investigations, accountability, security monitoring, and compliance. They can also help identify inappropriate or unusual access patterns.

Change histories are essential because technology systems evolve. Software may be updated, configurations changed, databases modified, permissions altered, and external services integrated. An audit trail should make it possible to understand significant changes and their timing.

Performance monitoring can provide evidence about whether the technology continues to operate within expected parameters. For AI systems, monitoring may include changes in accuracy, error rates, data distributions, drift, unusual outputs, or other relevant performance indicators.

Incident reports provide another important source of evidence. A mature audit system should allow organizations to connect incidents with relevant system configurations, data, users, controls, and response actions. This enables learning and supports continuous improvement.

Raji et al. (2020) argue for structured algorithmic auditing approaches that can examine systems beyond simple model-performance metrics. This perspective is particularly relevant to complex AI systems because model accuracy alone may not reveal problems involving data governance, deployment context, fairness, security, transparency, or organizational processes.

Auditability also supports regulatory and organizational assurance. When a regulator, auditor, internal review team, board, or other authorized evaluator examines a technology, sufficient evidence should exist to explain how the system was developed, deployed, operated, modified, and monitored.

Auditability must, however, be balanced with data protection and security. Creating extensive records may itself create privacy and cybersecurity risks. Audit information should therefore be appropriately protected, access-controlled, retained according to legitimate requirements, and disposed of securely when no longer needed.

The principle can be represented as:

System activity → evidence generation → secure record → traceability → independent review → finding → corrective action → documented follow-up.

Auditability therefore transforms technology from a system that merely produces outputs into a system whose important activities can be examined and reconstructed. This is especially important where decisions have significant consequences for individuals, organizations, public services, safety, finances, or other stakeholders.

9.23 Accountability

Accountability requires identifiable responsibility. A common problem with complex technology is the diffusion of responsibility:

Developer → Vendor → Organization → Operator → User

If something goes wrong, each actor may claim that another actor was responsible.

A robust evaluation system should therefore identify:

system owner;

technical owner;

data owner;

operational owner;

compliance owner;

risk owner; and

final decision authority.

Accountability is essential because increasingly complex technologies involve multiple participants. A technology may be designed by one organization, supplied by another, hosted by a third party, configured by an internal technical team, operated by employees, and used by customers. Without clear allocation of responsibility, failures can produce uncertainty about who should investigate, who should respond, who should compensate affected parties where applicable, and who has authority to change or suspend the system.

Accountability should begin before deployment. The organization should establish who has authority to approve the technology, who is responsible for ensuring that required evaluations are completed, who accepts residual risk, and who can authorize deployment.

The system owner should generally have responsibility for the overall purpose and operational status of the technology. The technical owner may be responsible for architecture, configuration, technical maintenance, and technical performance. The data owner may be responsible for data governance and access arrangements. The operational owner may oversee day-to-day use. The compliance owner may coordinate regulatory and policy requirements. The risk owner may be responsible for identifying, assessing, monitoring, and escalating material risks. Final decision authority should be clearly identified for consequential decisions.

These responsibilities may be assigned differently across organizations. The important principle is not that every organization must use identical titles but that responsibilities must be explicit and understood.

Accountability also requires authority. It is problematic to assign responsibility to an individual who does not have the authority or resources necessary to influence the outcome. Responsibility should therefore be accompanied by appropriate decision rights, access to information, resources, escalation mechanisms, and organizational support.

Accountability also requires evidence. Individuals and organizations should be able to demonstrate what decisions were made, why they were made, what evidence was considered, what controls were implemented, and how risks were monitored.

Third-party relationships create additional accountability challenges. Contracts should clarify responsibilities for security, data protection, system availability, incident reporting, maintenance, updates, service levels, audit cooperation, and termination. Outsourcing a technology function does not necessarily eliminate the organization's responsibility for managing the risks associated with that function.

Accountability should also extend to changes. When a system is materially modified, responsibility for evaluating the consequences of the change should be identified. This is particularly important for AI systems because changes in models, data, prompts, configuration, or external services may alter system behaviour.

Human oversight provides another accountability mechanism. Where technology supports consequential decisions, organizations should identify who is responsible for reviewing outputs and who has authority to override or reject them. Human involvement should not be merely symbolic; the responsible individual should have sufficient knowledge and authority to exercise meaningful judgment.

Accountability is also closely related to incident management. When an incident occurs, the organization should be able to determine who was responsible for detection, notification, containment, investigation, remediation, communication, and recovery. Clear accountability reduces delays and helps prevent responsibility from being lost across organizational boundaries.

The accountability structure can be represented as:

Purpose → ownership → responsibility → authority → evidence → oversight → review → corrective action.

This structure helps ensure that technology decisions remain connected to identifiable people and institutions rather than becoming attributed vaguely to “the system,” “the algorithm,” or “the technology.”

9.24 Incident Management

Technology evaluation does not end at deployment.

Organizations should establish procedures for:

detection;

reporting;

containment;

investigation;

communication;

remediation;

recovery; and

learning.

Incident information should feed back into the evaluation process.

Incident management is an essential component of technology governance because even well-designed technologies can experience failures, security events, operational disruptions, unexpected behaviours, or unintended consequences. The existence of an incident-management capability recognizes that risk cannot always be eliminated and that organizational resilience depends partly on how effectively incidents are handled.

Detection is the first stage. The organization must be capable of identifying that an abnormal event has occurred. Detection may depend upon automated monitoring, system alerts, user reports, cybersecurity tools, performance indicators, customer complaints, audit findings, or other sources of evidence. Detection should be timely enough to allow appropriate intervention.

Reporting establishes how an incident is communicated to the responsible individuals or teams. Employees should know what constitutes an incident, how to report it, and whom to contact. Reporting procedures should also allow escalation when an incident has potentially serious safety, security, legal, financial, environmental, or reputational consequences.

Containment seeks to prevent the incident from becoming more extensive. Depending upon the technology, containment may involve isolating a system, disabling an account, stopping a process, disconnecting a network, suspending a service, restricting access, or activating a safe operating mode. Containment decisions should be supported by predefined procedures where possible because delays can increase consequences.

Investigation seeks to determine what happened, why it happened, what systems and stakeholders were affected, and whether additional risks remain. Investigation should distinguish between immediate symptoms and underlying causes. A system failure may appear to result from a software error while deeper investigation reveals inadequate testing, poor change management, weak supplier controls, insufficient training, or architectural dependencies.

Communication is an important component of incident management. Relevant internal stakeholders need accurate information to coordinate response. External communication may also be required depending upon the nature of the incident, contractual obligations, legal requirements, regulatory requirements, and stakeholder interests. Communication should be factual and proportionate and should avoid creating additional confusion or misinformation.

Remediation addresses the underlying weakness that contributed to the incident. Remediation may involve software changes, security improvements, replacement of components, revised procedures, additional training, supplier changes, architectural redesign, policy modification, or strengthened governance.

Recovery concerns the restoration of normal or acceptable operations. Recovery should not simply mean restarting a system. It should include verification that systems are functioning correctly, data integrity has been restored where necessary, security weaknesses have been addressed, and stakeholders can safely resume normal operations.

Learning is the final but continuing stage. Incident information should be incorporated into future technology evaluation, risk registers, FMEA, Fault Tree Analysis, scenario analysis, cybersecurity assessments, training, governance, and design decisions. The purpose is not merely to close the incident but to reduce the probability or consequence of similar events in the future.

A useful incident-management sequence is:

Detection → reporting → classification → containment → investigation → communication → remediation → recovery → lessons learned → reassessment.

Incident classification can help determine the appropriate level of response. Minor operational incidents may be handled through routine procedures, whereas incidents involving safety, sensitive data, critical infrastructure, major financial loss, or significant service disruption may require senior management involvement and external notification where applicable.

Incident severity should therefore be evaluated according to the consequences and scope of the event. Relevant considerations may include the number of people affected, duration of disruption, sensitivity of information involved, financial consequences, safety implications, environmental effects, regulatory significance, reputational implications, and potential for further propagation.

Incident management should also be tested before a major incident occurs. Tabletop exercises, simulations, recovery exercises, communication drills, backup restoration tests, and cybersecurity response exercises can reveal weaknesses in procedures. An organization may discover during an exercise that contact information is outdated, responsibilities are unclear, recovery systems are incomplete, or decision authority is uncertain.

Incident records should be maintained appropriately because they provide evidence for future learning and assurance. Records may include timelines, affected systems, detection methods, response actions, decisions, communications, recovery activities, and corrective actions. Appropriate protection is necessary because incident records may contain sensitive security, personal, commercial, or technical information.

Incident management also contributes to organizational resilience. A resilient organization is not necessarily one that experiences no incidents. Rather, resilience includes the ability to anticipate, withstand, respond to, recover from, and learn from disruption. Incident management therefore provides a practical mechanism through which resilience can be strengthened over time.

Incident management should also extend to third-party incidents. If an external cloud provider, software supplier, payment processor, data provider, or telecommunications service experiences a disruption, the organization's own services may be affected. Contracts and governance arrangements should therefore establish how incidents involving external providers are detected, communicated, investigated, and resolved.

For AI and other adaptive technologies, incident management may require additional attention to model behaviour. An incident may involve inappropriate output, unexpected recommendations, data leakage, performance degradation, model drift, harmful automation, or misuse. Investigation may therefore need to consider the model version, relevant input data, configuration, prompts or parameters where applicable, human oversight, external dependencies, and operating context.

The feedback relationship between incident management and technology evaluation can be represented as:

Incident → evidence → root-cause analysis → risk reassessment → control improvement → technology modification → renewed testing → monitoring.

This creates a continuous improvement cycle rather than a closed incident-reporting process.

Ultimately, incident management demonstrates that technology evaluation is not a one-time activity. A technology should be evaluated before deployment, during implementation, throughout operation, after significant changes, and following incidents. Real-world incidents provide evidence about assumptions that may have been incorrect, controls that may have been ineffective, and risks that may not have been fully anticipated.

The information generated through incident management should therefore feed directly back into the broader technology evaluation framework. FMEA can be updated with newly observed failure modes. Fault trees can be revised when new causal pathways are identified. Bow-Tie Analysis can be modified when preventive or mitigation barriers prove inadequate. Scenario assumptions can be reassessed. Cybersecurity controls can be strengthened. Data-governance procedures can be revised. Accountability structures can be clarified. Security-by-design principles can be incorporated into subsequent system versions.

In this way, incident management completes an important part of the technology-risk lifecycle:

Identify → assess → design controls → deploy → monitor → detect → respond → recover → learn → redesign → reassess.

The central principle is that an incident should not be treated solely as an isolated operational failure. Properly analysed, it becomes a source of organizational and technological knowledge. The objective is not simply to restore the system to its previous state but, where appropriate, to use evidence from the incident to improve the technology, strengthen controls, clarify accountability, enhance resilience, and reduce the likelihood or consequences of recurrence.

9.25 Case Study: Autonomous Warehouse Technology

Consider an autonomous warehouse system using robots to move inventory.

Technical benefits may include increased throughput, continuous operation and improved inventory tracking.

However, evaluation should also consider:

collision risk;

worker safety;

cybersecurity;

software failure;

maintenance;

employment implications;

data collection;

vendor dependency;

energy consumption;

emergency procedures; and

organizational readiness.

A technically successful pilot therefore does not automatically establish organizational readiness for full-scale adoption.

9.26 Technology Risk Evaluation Matrix

9.27 Technology Risk Evaluation Protocol

A practical protocol should include:

Stage 1: identify the technology.

Stage 2: identify stakeholders.

Stage 3: identify hazards and threats.

Stage 4: identify vulnerabilities.

Stage 5: estimate consequences.

Stage 6: identify controls.

Stage 7: assess residual risk.

Stage 8: determine risk acceptance criteria.

Stage 9: establish monitoring requirements.

Stage 10: define reassessment triggers.

9.28 Research Questions

How can organizations evaluate emerging technology risk when empirical evidence is limited?

How does cybersecurity risk influence technology adoption?

How can organizations integrate technical, operational and regulatory risk?

How effective are technology risk registers in managing emerging technologies?

How can governance structures improve accountability for technology-related failures?

9.29 Chapter Summary

Technology risk is multidimensional. Technical reliability is necessary but insufficient. Effective evaluation must also examine cybersecurity, operational processes, privacy, legal requirements, financial exposure, environmental effects, human factors and governance.

Risk management should be continuous because technology and its operating environment change over time.

The AMTEF therefore treats risk and governance as dimensions that intersect with all other technology evaluation dimensions rather than as separate final-stage checks.

9.30 Discussion Questions

Why is technical reliability insufficient for determining whether a technology is acceptable?

How can organizations distinguish risk from uncertainty?

What are the limitations of quantitative probability estimates for emerging technologies?

How should cybersecurity be incorporated into technology evaluation?

What makes a technology auditable?

Who should be accountable when an AI-enabled system causes harm?