Chapter 6. Privacy, Data Exploitation, and Surveillance

6.1 Data can travel farther than users expect

AI tools may process prompts, uploaded documents, images, voice recordings, meeting transcripts, code, and other personal or confidential material. Depending on the service, settings, contract, and jurisdiction, information may be retained, reviewed, logged, or used to improve a service. Users should not assume that a conversation is automatically confidential simply because it feels private.

Sensitive information includes identity documents, passwords, financial details, health information, student records, unpublished research, customer lists, confidential contracts, and private communications. Uploading such material to an unapproved service can create risks even when no immediate breach is visible.

6.2 Data minimisation and purpose limitation

Data minimisation means collecting and sharing only what is necessary for a defined purpose. Before using AI, remove direct identifiers where possible, redact irrelevant details, use synthetic examples for demonstrations, and check whether the service is approved for the data category. Pseudonymisation reduces direct identification but does not guarantee anonymity; combinations of details can sometimes re-identify a person.

An organisation should document what data enters the system, why it is needed, where it is processed, how long it is retained, who can access it, whether it is transferred to another provider, and how it can be deleted. These questions belong in procurement and governance—not only in the hands of individual employees.

6.3 Surveillance and power imbalance

AI can support legitimate functions such as detecting equipment failure or improving accessibility. It can also enable intrusive monitoring of workers, students, customers, or public spaces. Continuous behavioural scoring can chill expression, intensify stress, and encourage people to optimise for what a system measures rather than what matters.

Biometric identification, emotion inference, behavioural prediction, and location analysis require particular caution. An inferred emotion or personality label may be unreliable and can become harmful when treated as fact. Institutions should examine necessity, proportionality, alternatives, legal requirements, notice, consent where applicable, and routes to challenge the result.

6.4 A privacy checklist

• Is the tool approved for the type of information being processed?
• Can the task be completed with less data or a synthetic example?
• Have identifiers, secrets, and irrelevant details been removed?
• Are retention, training use, access, transfer, and deletion terms understood?
• Is the data subject entitled to notice, consent, access, correction, or another remedy under applicable law?
• Has a privacy or security specialist reviewed high-risk processing?