Chapter 11. High-Stakes AI: Health, Finance, Justice, and Public Services
11.1 Why context changes the standard
A low-stakes suggestion can be corrected with little harm. A system that influences diagnosis, treatment, credit, insurance, immigration, policing, education access, or public benefits can affect rights and life chances. High-stakes use requires stronger evidence, validation, oversight, transparency, security, and routes for redress.
An AI score should not be mistaken for a complete account of a person. Data can be missing, outdated, incorrectly matched, or collected under unequal conditions. A model may identify statistical association without establishing causation or explaining what action is justified.
11.2 Health and well-being
AI may support clinical documentation, image analysis, triage, research, and patient education. Risks include inaccurate advice, missed symptoms, automation bias, privacy leakage, poor performance on under-represented populations, and unclear responsibility. People should not use a general-purpose chatbot as a substitute for professional diagnosis or emergency care. Clinicians and health organisations should validate systems for their intended setting and preserve professional judgement.
Patients should ask what role AI played, whether a qualified professional reviewed the result, how personal data is handled, and how to seek a second opinion or correct an error. Providers should communicate limitations in language patients can understand.
11.3 Finance, credit, and insurance
AI may help detect unusual transactions, forecast demand, or support fraud review. It may also create false positives, reproduce historical exclusion, or make it difficult for customers to understand why a decision was reached. Financial institutions should test performance under changing economic conditions, assess disparate impacts where lawful, monitor model drift, and provide appropriate explanations and complaint procedures.
Generative AI can also create persuasive but false financial claims. Verify the identity of advisers, the status of firms, and the underlying documents through official channels. Do not rely on synthetic endorsements, urgency, or polished presentation as proof of legitimacy.
11.4 Justice and public administration
Automated tools used in justice or public services can affect liberty, eligibility, enforcement, or access to support. Such uses require strict attention to due process, legal authority, evidentiary standards, bias, explainability, and contestability. A person affected by a decision should not be left with “the computer said so” as the only explanation.
Public institutions should publish clear descriptions of significant AI uses, identify accountable officials, conduct impact assessments, maintain records, and provide accessible ways to challenge errors. Procurement should include independent testing and meaningful oversight rather than relying solely on vendor assurances.
11.5 A high-stakes decision rule
If an AI output could materially affect someone’s rights, safety, livelihood, health, access to education, or essential services, do not allow an unreviewed model output to be the final authority. Require documented evidence of fitness for purpose, qualified human review, appeal, monitoring, and an accountable decision-maker.