Chapter 3. Generative AI, Large Language Models, and Multimodal Systems
3.1 What generative AI does
Generative AI produces content based on patterns learned during training and, in some systems, information supplied at the time of use. A text model can draft an email, explain a concept, propose a plan, or transform a passage into a different style. Image models can create visual concepts; audio systems can transcribe or generate speech; video systems can create or edit moving images. Capabilities vary across products and change rapidly.
Generated content is not necessarily copied verbatim from a source, nor is it guaranteed to be original, correct, or legally unrestricted. Users should check factual claims, attribution, privacy, copyright, and the terms governing commercial use.
3.2 Large language models
Large language models (LLMs) are trained to process and generate language. Many use transformer architectures, which rely on attention mechanisms to represent relationships between tokens. During generation, the model estimates plausible next tokens conditioned on prior tokens and the available context. This process can produce coherent explanations, but fluency should not be mistaken for verified knowledge.
An LLM may generate an answer that sounds confident but contains an invented reference, inaccurate quotation, incorrect date, or unsupported detail. This behaviour is commonly called a hallucination. It can occur because the model is optimising for plausible continuation rather than consulting a guaranteed source of truth. Retrieval, tool use, constrained output, and verification can reduce some errors but cannot eliminate them.
3.3 Context windows, memory, and retrieval
A context window is the amount of material a model can consider within a particular interaction. A longer context can support larger documents, but it does not ensure that every detail will be used correctly. Product memory, conversation history, and enterprise data connections are separate features and differ by service.
Retrieval-augmented generation (RAG) connects a model to selected documents or databases. The system retrieves relevant passages and uses them to help generate a response. RAG can improve grounding when the source collection is current and reliable. It still needs source selection, permission controls, citations, and checks that the answer accurately reflects the retrieved material.
3.4 Multimodal AI
Multimodal systems can work with more than one kind of input or output, such as text, images, audio, video, or structured data. A user might ask a system to describe a diagram, summarise a meeting recording, or explain a chart. Such systems can improve accessibility and workflow speed, but may misread labels, miss context, misidentify objects, or overlook important details.
Image, audio, and video outputs can also be synthetic. In consequential contexts, provenance and verification matter. A convincing visual or voice recording is not, by itself, proof that an event occurred or that a person made a statement.
3.5 AI agents and tool use
An AI agent is a system designed to pursue a goal through a sequence of steps, often using tools such as search, code execution, databases, calendars, or business applications. Some agents can plan, act, inspect results, and revise their approach. This can be useful for bounded workflows, but increases the importance of permissions, logging, testing, and limits on actions.
An agent that can draft a report has a different risk profile from one that can send messages, modify customer records, issue refunds, or deploy software. Begin with read-only or approval-based access. Give the system only the permissions it needs, require human confirmation for consequential actions, and provide a way to stop or reverse actions.
3.6 Fine-tuning and customisation
Fine-tuning adjusts a model using additional examples to influence its behaviour for a particular task or domain. Prompt templates, retrieval, structured outputs, and workflow design are often cheaper and easier first steps. Fine-tuning is not a substitute for current facts, quality data, privacy controls, or evaluation.
Customisation should be driven by a documented performance need. A team should compare a customised approach with a baseline and assess whether the gains justify the cost, maintenance, and added complexity.