Chapter 2. How AI Works—and Why It Can Fail

2.1 Training is not the same as understanding

Many modern AI systems learn statistical relationships from examples. A model adjusts internal parameters during training to perform a task according to an objective. A language model generates text by estimating likely continuations from context; an image model learns patterns that help it generate or interpret visual content. These processes can produce impressive results without guaranteeing that every output is true, grounded, fair, or appropriate.

Performance depends on training data, model architecture, optimisation, testing, deployment conditions, and the prompt or input. A system that performs well on a benchmark may fail on a new population, language, document type, or unusual situation. Benchmark scores are useful evidence but do not replace testing in the environment where the system will actually be used.

2.2 Common technical limitations

Distribution shift: real-world inputs differ from the data or conditions on which a system was developed.
Confabulation: generative systems may produce plausible but unsupported claims or invented references.
Opacity: it may be difficult to explain why a complex model produced a particular output.
Calibration problems: the system’s confidence-like language or score may not correspond reliably to its actual accuracy.
Context loss: a model may miss a key qualification, exception, or detail in a long prompt or document.
Automation and tool errors: a connected AI agent may misinterpret instructions, call the wrong tool, or take an action that is hard to reverse.
Data contamination or leakage: evaluation results can be misleading if test data overlaps with training data or if sensitive information is exposed.
Uneven performance: accuracy can vary across languages, dialects, demographic groups, accessibility needs, and less-represented contexts.

2.3 Why fluent output can mislead

People often interpret clear, polished language as evidence of expertise. In generative AI, fluency is not a guarantee of correctness. A response can be coherent, detailed, and wrong. The system may also blend true statements with errors, making the answer harder to evaluate than an obviously nonsensical result. NIST’s generative AI profile identifies confabulation, privacy, information security, harmful bias, and other risks as areas requiring deliberate management (NIST, 2024).

Treat output as a draft, hypothesis, or recommendation until it has been checked to a level appropriate for the stakes. For factual claims, look for primary sources. For calculations, recalculate independently. For legal or medical questions, consult qualified professionals. For important decisions, record what evidence was used and who approved the outcome.

2.4 AI agents and increased consequence

An AI agent is a system that can pursue a goal through multiple steps, sometimes using tools, files, databases, or external services. A tool-connected agent may do more than suggest text: it may prepare a transaction, modify a document, send a message, or change a system. As autonomy and permissions increase, the consequences of error increase too. Apply least-privilege access, restricted actions, human approval for consequential steps, logs, and a tested stop mechanism. Do not give an experimental agent broad access merely because a demonstration looks impressive.