What is AI privacy?

AI privacy concerns what AI systems know about people and what they can reveal. Three risks dominate. Models memorize fragments of training data and can be made to reproduce them. Models infer sensitive attributes that were never supplied. And personal data collected for one purpose is reused to train systems the subject never agreed to. Real attacks show how this plays out, such as the Imprompter attack on chatbots, and the obligations sit mainly in privacy law, with the EU AI Act layered on top.

What AI privacy covers

Privacy law protects personal information: anything about an identifiable individual, from name and email to location, biometrics and behavioural patterns. AI changes the problem in four ways:

• Scale. Training sets are scraped or licensed in volumes no one can review record by record.
• Persistence. Data absorbed into model weights cannot simply be found and deleted like a row in a database.
• Inference. Models generate new personal information (predictions, profiles, scores) that the individual never provided.
• Interaction. Chatbots and agents collect personal data through prompts, uploads and connected tools, often more than users realize.

AI privacy is therefore wider than data protection compliance for a training set. It covers the whole lifecycle: what goes in, what the model retains, what it can infer, what it outputs, and what users type into it.

The privacy risks specific to AI

Some risks are familiar, such as breaches and unauthorized access to large datasets. The ones below are specific to how AI systems work:

• Memorization and extraction. Models reproduce verbatim fragments of their training data. Carlini et al. (2021) extracted names, phone numbers and email addresses from GPT-2. Nasr et al. (2023) showed that a simple "divergence" prompt made ChatGPT emit training data at many times its normal rate, and that larger, more capable models memorize more.
• Membership inference. An attacker tests whether a specific person's record was in the training set, which can itself be sensitive (a model trained on a clinic's patients). It is a proven risk for smaller models trained on specific datasets. For large language models trained once over huge corpora, Duan et al. (2024) found current attacks barely beat random guessing, so vendor claims either way warrant scrutiny.
• Attribute inference. Staab et al. (2023) showed that large language models could infer location, income, age and other attributes from Reddit posts with accuracy close to human investigators, at a fraction of the cost. Nothing was "breached"; the model worked it out.
• Re-identification. Combining de-identified datasets, or querying a model about rare individuals, can reconnect data to a person. Anonymization that held for a static table often fails against a model with broad background knowledge.
• Purpose creep. Data collected to deliver a service, such as support chats, documents or photos, is reused to train models. That is the core consent question in most regulator actions against AI developers.
• Prompt and agent leakage. Staff paste personal data into public chatbots, and agents with tool access can be manipulated into sending it elsewhere. Imprompter used an obfuscated prompt to make chatbots quietly exfiltrate personal details from a conversation through a hidden image link.
• Inaccurate personal data. A model that "hallucinates" false facts about a named person creates inaccurate personal information, which access and correction rights were not designed for.
• Discrimination and opacity. Decisions built on personal data can be unfair (see bias in AI systems) and hard to explain or challenge (see explainable AI).

Privacy versus utility

AI systems need data, and some use cases need personal data to work at all: fraud detection, clinical decision support, personalization. The trade-off is real but often narrower than assumed. Most training value comes from patterns, not from identifying who a record belongs to. That is why minimization, de-identification and privacy-enhancing technologies often cost less accuracy than expected.

The balance is context-specific and has to be re-evaluated as a system, its data and its users change. The practical test is a documented one: a privacy impact assessment that states the purpose, the data needed for it, the residual risk, and who accepted that risk.

Protecting privacy in AI systems

Transparency

People cannot exercise rights over processing they do not know about. Transparency for AI means saying, in plain language, what personal data trains or feeds the system, where it came from, what the system is used for, whether decisions are automated, and how to object, access or correct. The Canadian ChatGPT investigation turned partly on this: users were not told clearly that "publicly accessible" sources included social media and forums. See AI transparency.

Consent and control

Consent remains the default legal basis in Canada, and it is hard to obtain for web-scraped training data. Control means meaningful choices: opting out of training on one's chats, objecting to processing, and deleting or correcting data. For models, deletion is the hard part. "Machine unlearning" is an active research area but not yet a reliable way to remove an individual from trained weights, so the realistic controls are filtering before training, output-level suppression, and retraining on clean data over time.

Technological measures

Privacy-enhancing technologies (PETs) reduce what a system exposes:

• Encryption. Protects data in transit and at rest; confidential computing extends this to data in use inside trusted execution environments.
• De-identification and anonymization. Removing or masking identifiers before training. PII filtering of training corpora is now standard at major developers, and it was one of the fixes OpenAI made after the Canadian investigation.
• Differential privacy. Adds calibrated noise so that no single record measurably changes the output. It is the only PET with a mathematical guarantee against memorization, at some cost in accuracy. NIST published guidance on evaluating its claims in SP 800-226 (2025).
• Federated learning. Trains across devices or institutions without pooling raw data. It reduces central exposure but does not by itself prevent leakage through model updates, so it is usually combined with differential privacy or secure aggregation.
• Synthetic data. Generated records that preserve statistical properties. Useful for testing and sharing, but synthetic data can still leak information about real individuals if generated carelessly.
• Output controls. Filters that block personal data about private individuals from model responses, plus logging and rate limits that make extraction attacks harder.

Regulatory framework and policy implications

European Union: GDPR and the AI Act

The GDPR applies to AI whenever personal data is processed, in training or in use. Its principles of lawfulness, purpose limitation, minimization, accuracy, storage limitation, security and accountability all bite. Key interpretations:

• EDPB Opinion 28/2024 (December 2024): a model trained on personal data is not automatically anonymous; legitimate interest can be a legal basis for development and deployment only after a documented three-step test (interest, necessity, balancing), not as a default; and unlawful processing during development can taint the lawfulness of later use of the model.
• Relative approach to personal data. In EDPS v SRB (C-413/23 P, September 2025), the Court of Justice held that pseudonymized data may not be personal data for a recipient who has no reasonable means to re-identify individuals, while it remains personal data for the party holding the key.
• Automated decisions. Article 22 restricts decisions based solely on automated processing that have legal or similarly significant effects. The CJEU has read it broadly, including credit scores that a lender relies on heavily.
• EU AI Act. Adds data governance duties for high-risk systems (Article 10), transparency for chatbots and synthetic content, and a ban on untargeted scraping of facial images to build recognition databases. The Digital Omnibus on AI, in force since 27 July 2026, moved most high-risk obligations to 2 December 2027 (Annex III) and 2 August 2028 (Annex I).

The Digital Omnibus changes to the GDPR are not law. The Commission's November 2025 proposal would recognize AI development and operation as a legitimate interest (a new Article 88c, with safeguards including an unconditional right to object) and narrow the definition of personal data to what a given entity can identify. As of October 2026, Parliament's committees were still working through more than 1,750 amendments, member states had not agreed a Council mandate, and the Council's compromise text had dropped the new personal data definition.

Canada

• PIPEDA still governs private-sector privacy federally, with consent as the default basis. The Office of the Privacy Commissioner (OPC) and its provincial counterparts set out principles for generative AI in December 2023.
• The OpenAI investigation. In PIPEDA Findings #2026-002 (6 May 2026), the OPC and the Quebec, British Columbia and Alberta regulators found that OpenAI's initial training of ChatGPT involved overcollection, lacked valid consent and transparency, did not adequately address inaccurate outputs about people, and was released without full accountability for known risks. The federal complaint was well-founded and conditionally resolved. British Columbia and Alberta found their consent rules still breached, and Quebec found consent unresolved.
• Quebec Law 25. Since September 2023, section 12.1 of Quebec's private-sector privacy act requires organizations to tell people when a decision about them is made exclusively by automated processing, and on request to explain the personal information, reasons and principal factors used, with a right to have the decision reviewed by a person. Privacy impact assessments and privacy-by-default settings also apply.
• Bill C-27 is dead; Bill C-36 is pending. C-27 (the Consumer Privacy Protection Act and AIDA) died on prorogation in January 2025. On 15 June 2026, the government tabled Bill C-36, the Protecting Privacy and Consumer Data Act. It would move enforcement to a new Digital Safety and Data Protection Commission with penalties of up to C$10 million or 3% of global revenue, add a legitimate-interest exception, a deletion right, and definitions of de-identified and anonymized data, and require explanations of automated decision systems that have significant effects. It contains no AI-specific regime like AIDA, and it was not law as of October 2026.

United States

• No federal privacy statute (as of October 2026). The FTC uses its unfair and deceptive practices authority, including orders requiring companies to delete models trained on improperly obtained data, and a facial recognition ban against Rite Aid (2023).
• State laws. As of October 2026, about 20 states have comprehensive consumer privacy laws, with Indiana, Kentucky and Rhode Island taking effect on 1 January 2026. Most give a right to opt out of profiling used for decisions with legal or similarly significant effects and require data protection assessments for high-risk processing.
• California ADMT rules. Regulations approved in September 2025 under the CCPA took effect on 1 January 2026. Risk assessment duties apply from that date, with first submissions to the California Privacy Protection Agency due by 1 April 2028. Pre-use notice, opt-out and access rights for automated decision-making technology used for significant decisions apply from 1 January 2027.
• Colorado. The original Colorado AI Act (SB 24-205) never took effect: in xAI v. Weiser (D. Colo.), a 27 April 2026 order barred its enforcement pending a preliminary-injunction ruling. Its replacement, SB 26-189, signed 14 May 2026, is a narrower notice, explanation and human-review regime for consequential decisions, operative 1 January 2027.

Other jurisdictions

The UK GDPR, Singapore's PDPA, Brazil's LGPD and many others apply similar principles to AI. Global programs usually build to the GDPR and Quebec standards and adjust from there.

Role of governance in AI privacy

Privacy controls for AI are built into the AI governance program rather than applied as a legal sign-off at launch:

• Inventory. Know which AI systems process personal data, including vendor tools and staff use of public chatbots. An AI acceptable use policy sets what may be entered.
• Impact assessment. Run a privacy impact assessment for each AI use that touches personal data, covering training data provenance, inference risk and automated decisions. Align it with the AI system impact assessment under ISO/IEC 42001.
• Vendor terms. Confirm in contract whether the organization's data trains the vendor's models, where it is processed, how long it is kept, and how access and deletion requests flow.
• Rights handling. Decide how access, correction and deletion requests that touch a model will be answered, before the first one arrives.
• Testing. Include extraction and leakage tests in red teaming for systems trained or fine-tuned on personal data.
• Ownership. Name an accountable owner for each system. See AI accountability.

Internal policy, external regulation and multi-stakeholder standards (ISO, NIST, industry codes) all play a part, but only internal governance produces evidence about an organization's own systems.

Where AI privacy regulation is heading

Three trends are visible as of October 2026:

• AI-specific rules through privacy law. Rather than standalone AI acts, Canada (Bill C-36), California (ADMT rules) and Colorado (SB 26-189) are regulating AI through notice, explanation and human review of automated decisions.
• Accountability for training data. Regulators in Canada, the EU and the US now treat how a model was trained as a live compliance question, with remedies ranging from commitments to deletion of models.
• Transparency mandates. Disclosure duties for chatbots, synthetic content and automated decisions are spreading, led by Article 50 of the EU AI Act.

The open question is the EU's direction. If the GDPR strand of the Digital Omnibus passes in something like the Commission's form, legitimate interest for AI training becomes easier in Europe just as Canadian regulators are insisting on consent.

Summary

AI privacy is not a separate discipline from data protection; it is data protection applied to systems that memorize, infer and act. Effective programs know where personal data enters their AI systems, can explain what the systems do with it, and can answer a regulator's question about training data with documents rather than assurances.

Related references

For the program that privacy controls sit inside, see what AI governance is and the AI acceptable use policy. For how privacy is tested, see AI audit. For human review of automated decisions, see human-in-the-loop oversight. For attacks that exfiltrate personal data, see AI attacks and defences.

Sources and references

• Regulation (EU) 2016/679 (GDPR): https://eur-lex.europa.eu/eli/reg/2016/679/oj
• EDPB Opinion 28/2024 on AI models (17 December 2024): https://www.edpb.europa.eu/our-work-tools/our-documents/opinion-board-art-64/opinion-282024-certain-data-protection-aspects_en
• CJEU, EDPS v SRB, C-413/23 P (4 September 2025): https://curia.europa.eu/juris/liste.jsf?num=C-413/23
• European Commission, Digital Omnibus Regulation proposal (19 November 2025): https://digital-strategy.ec.europa.eu/en/library/digital-omnibus-regulation-proposal
• European Parliament Legislative Train, Digital package: https://www.europarl.europa.eu/legislative-train/theme-a-new-plan-for-europe-s-sustainable-prosperity-and-competitiveness/file-digital-package
• Regulation (EU) 2026/1744 (Digital Omnibus on AI): https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=OJ:L_202601744
• OPC, PIPEDA Findings #2026-002, joint investigation of OpenAI (6 May 2026): https://www.priv.gc.ca/en/opc-actions-and-decisions/investigations/investigations-into-businesses/2026/pipeda-2026-002/
• OPC, Principles for responsible, trustworthy and privacy-protective generative AI technologies: https://www.priv.gc.ca/en/privacy-topics/technology/artificial-intelligence/gd_principles_ai/
• Quebec, Act respecting the protection of personal information in the private sector (P-39.1): https://www.legisquebec.gouv.qc.ca/en/document/cs/P-39.1
• Parliament of Canada, Bill C-36 (45th Parliament, 1st session): https://www.parl.ca/legisinfo/en/bill/45-1/c-36
• California Privacy Protection Agency, regulations: https://cppa.ca.gov/regulations/
• Colorado SB 26-189: https://leg.colorado.gov/bill_files/116236/download
• Carlini et al., "Extracting Training Data from Large Language Models" (2021): https://arxiv.org/abs/2012.07805
• Nasr et al., "Scalable Extraction of Training Data from (Production) Language Models" (2023): https://arxiv.org/abs/2311.17035
• Duan et al., "Do Membership Inference Attacks Work on Large Language Models?" (2024): https://arxiv.org/abs/2402.07841
• Fu et al., "Imprompter: Tricking LLM Agents into Improper Tool Use" (2024): https://arxiv.org/abs/2410.14923
• Staab et al., "Beyond Memorization: Violating Privacy via Inference with Large Language Models" (2023): https://arxiv.org/abs/2310.07298
• NIST SP 800-226, Guidelines for Evaluating Differential Privacy Guarantees: https://csrc.nist.gov/pubs/sp/800/226/final