Chapter 10. Security, Reliability, Misinformation, and AI Failure
10.1 Common failure modes
AI systems can produce fabricated facts, misread inputs, follow misleading instructions, expose sensitive information, reproduce bias, fail on rare cases, or behave differently after an update. Failures may arise from the model, the data, the interface, integration, human procedures, or the broader operating environment.
A useful incident review asks what happened, which safeguards failed, who was affected, whether the output was detected, how harm was limited, and what changes will prevent recurrence. Blaming an individual user alone may overlook process or design weaknesses.
10.2 Cybersecurity and prompt injection
AI connected to external documents, websites, email, or tools can encounter malicious instructions embedded in content. Prompt injection is an attempt to manipulate a model by placing instructions in its inputs or retrieved material. Systems should treat retrieved content as data, not automatically as trusted instructions.
Security controls include least-privilege access, separation between instructions and untrusted content, validation of tool inputs, restricted network access, logging, sandboxing, rate limits, secret management, and human approval for consequential actions. Test both expected use and plausible misuse before deployment.
10.3 Data leakage and confidential information
An AI tool may retain prompts, files, or outputs according to its service design and terms. Even if a provider does not use data for model training, the service may still process or store data to provide the product. Organisations must understand the actual settings and contractual terms rather than relying on general assumptions.
Do not paste passwords, authentication tokens, private keys, or confidential records into a chatbot. Use approved secure workflows and report accidental disclosure promptly through the organisation's incident process.
10.4 Deepfakes and misinformation
Generated text, voice, images, and video can be used to create misleading content. Verification should consider provenance, the original source, corroborating evidence, contextual consistency, and independent reporting. Detection tools can help but may make mistakes, especially as generation techniques change.
Before sharing a dramatic claim, verify whether the source is authentic and whether reliable independent sources support the claim. A visual that looks real is not conclusive evidence. Organisations should establish procedures for handling suspected impersonation, manipulated media, and false statements attributed to staff.
10.5 Reliability controls
Controls include test sets, adversarial testing, restricted output formats, source citations, retrieval from approved knowledge bases, independent calculation, human review, monitoring, incident response, and rollback. No single control is sufficient for every use case.
High-risk systems need documented acceptance criteria and a safe fallback. If the model is unavailable or fails validation, the workflow should pause or revert to an approved manual process rather than silently producing unverified results.
10.6 Environmental and social costs
AI workloads use computing resources and can contribute to electricity demand, water use, hardware production, and electronic waste. The impacts differ by model, data centre, energy source, location, and workload. Organisations should consider whether a smaller model, fewer repeated calls, caching, or conventional automation can achieve the same goal with lower cost and resource use.