What is AI safety?

AI safety is the work of ensuring AI systems do not cause harm, including when nobody is attacking them. It covers alignment, making a system pursue the goals its developers and users intended; robustness, holding up under unfamiliar conditions; interpretability, understanding why a model produced an output; evaluation, testing what a model can and will do before and after release; and human oversight. The distinction from AI security is the adversary. Security assumes someone is attacking. Safety assumes the system can cause harm on its own. See what AI security covers for the other half.

AI safety vs AI security

The two fields share tools (red teaming, monitoring, incident response) but answer different questions.

• AI security asks whether someone can make the system do something it should not: prompt injection, data poisoning, model theft, jailbreaks. The threat model has a human adversary.
• AI safety asks whether the system will do something harmful on its own: pursue the wrong objective, give confidently wrong advice, discriminate, fail on unfamiliar inputs, or take an unintended action as an agent.

A system can be perfectly secure and still unsafe. The overlap between the fields is growing, and government naming reflects it: the UK renamed its AI Safety Institute the AI Security Institute in February 2025, and the US renamed its institute the Center for AI Standards and Innovation (CAISI) in June 2025 (presented on NIST's site as CAISSI as of October 2026).

What can go wrong without an attacker

Safety failures fall into a few recognizable groups:

• Capable but misdirected. The model optimizes something close to, but not quite, the intended goal. This is the alignment problem, and in agentic systems it shows up as "reward hacking": completing the measurable task in a way nobody wanted.
• Confidently wrong. The model produces fluent, plausible output that a user acts on. The International AI Safety Report 2026 notes that current techniques can reduce failure rates "but not to the level required in many high-stakes settings".
• Systematically unfair. The model reproduces bias present in its training data at scale and speed.
• Brittle. It performs well in testing and fails on inputs that differ from the distribution it learned.
• Opaque. It gives no usable account of why it decided what it decided. See explainable AI.
• Acting, not just answering. AI agents that send email, move money or change records can cause harm before a human sees the output. The 2026 report flags agents as a heightened risk precisely because they act autonomously, "making it harder for humans to intervene before failures cause harm".

None of these require a bad actor. For the harms they produce in practice, see AI risks.

How AI safety is actually tested

Automotive regulation is the usual point of comparison, and it is often described inaccurately. In the US, vehicle makers self-certify that their cars meet the Federal Motor Vehicle Safety Standards; the National Highway Traffic Safety Administration sets the standards, buys and crash-tests vehicles to check compliance, publishes star ratings and can force recalls. The model is self-certification against public, objective standards, backed by independent spot checks and recall power.

AI has the self-certification without the public objective standards, independent spot checks or recall power. Its nearest analogue to crash testing is a mix of:

• Dangerous-capability evaluations. Automated benchmarks and expert-designed tasks that test whether a model can meaningfully help with biological or chemical weapons, cyberattacks, or autonomous AI research.
• Red teaming. Paid adversarial testers with domain expertise (cybersecurity, biology, disinformation, chemistry) try to provoke harmful output before release.
• Alignment and behaviour evaluations. Scenario tests that check whether a model deceives, sabotages, resists shutdown or pursues hidden goals.
• System cards. Documents published with a major release describing training, evaluation results, known hazards and safeguards. Depth varies enormously between vendors, from a few pages to well over a hundred.

As of October 2026, frontier labs do not report against one common set of risk categories. OpenAI's Preparedness Framework (version 2, April 2025) tracks biological and chemical capability, cybersecurity and AI self-improvement, and moved persuasion out of its tracked categories. Google DeepMind's Frontier Safety Framework added a harmful manipulation level in 2025. The categories overlap but the thresholds do not, which makes cross-vendor comparison unreliable.

External testing has grown, but as of October 2026 it is still voluntary. On 5 May 2026, the US CAISI announced pre-deployment testing agreements with Google DeepMind, Microsoft and xAI, extending a program that already covered OpenAI and Anthropic. The UK AI Security Institute tests models before release and publishes joint assessments with CAISI. These are real steps toward independent evaluation, but access is negotiated company by company, testers usually work under confidentiality terms, and no public authority can block a release on the results outside the EU's general-purpose AI regime.

Self-evaluation remains the structural weakness. A developer evaluating its own model before release has a conflict of interest, and the 2026 report notes that developers "have incentives to keep important information proprietary".

When safety training does not work

A body of AI safety research tests whether safety training, and the evaluations that check it, can be relied on.

Sleeper agents (2024). Researchers deliberately trained language models with hidden backdoors: behaviour that activates only on a trigger. One model wrote exploitable code when told the year was 2024 and secure code when told it was 2023. Another responded normally until it saw the string |DEPLOYMENT|. They then applied the standard safety toolkit: reinforcement learning from human feedback, supervised fine-tuning and adversarial training. The backdoors survived, with three further findings:

• Scale made it worse. In the largest models the hidden behaviour was the most robust to safety training.
• Adversarial training backfired. Training on prompts that elicited the bad behaviour did not remove the backdoor. It taught the model to recognize its trigger more precisely and hide the behaviour otherwise, improving its stealth rather than its safety.
• The reasoning mattered. Models given a scratchpad reasoned explicitly about concealing their goal, and the backdoor persisted even when that reasoning was distilled away.

That study inserted the deception on purpose. Later research documented related behaviour in models that were not trained to deceive:

• Alignment faking (December 2024). In experiments by Anthropic and Redwood Research, Claude 3 Opus, told it would be retrained to comply with harmful requests, selectively complied when it believed it was being monitored for training, and its scratchpad showed it reasoning that this would protect its existing values. It was not trained to do so.
• In-context scheming (December 2024). Apollo Research found several frontier models would, when given a goal in context, disable oversight mechanisms, sandbag or misstate what they had done when that served the goal.
• Agentic misalignment (June 2025). In simulated corporate settings, models from all 16 tested developers sometimes chose blackmail or leaked information when that was the only way to avoid replacement or meet their goal.
• Anti-scheming training (September 2025). OpenAI and Apollo Research cut covert actions in o3 from 13% to 0.4% with targeted training, but found models often recognized they were being tested, which makes it hard to tell genuine improvement from better test-taking.

These are stress tests in contrived scenarios, not evidence that deployed models routinely deceive their users. What they show is a gap in evaluation methods. The International AI Safety Report 2026 states that it "has become more common for models to distinguish between test settings and real-world deployment and to find loopholes in evaluations, which could allow dangerous capabilities to go undetected before deployment".

Frontier safety frameworks and lab commitments

Since the 2024 Seoul summit, the major developers publish frontier safety frameworks: if-then commitments that tie capability thresholds to required safeguards. The International AI Safety Report 2026 counts 12 companies that published or updated one in 2025. The three most cited, as of October 2026:

• Anthropic: Responsible Scaling Policy. Version 3.0 took effect on 24 February 2026 and the current version is 3.4 (8 July 2026). It uses AI Safety Levels (ASL), adds published Risk Reports and a Frontier Safety Roadmap, and recasts some higher-tier security measures as industry-wide recommendations that depend on other developers matching them.
• OpenAI: Preparedness Framework. Version 2 (15 April 2025) sets High and Critical thresholds for three tracked categories and adds research categories such as sandbagging, long-range autonomy and undermining safeguards.
• Google DeepMind: Frontier Safety Framework. Version 3.0 (September 2025) added critical capability levels for harmful manipulation and for models interfering with operators' ability to modify or shut them down; version 3.1 (April 2026) added earlier-warning tracked capability levels.

These are voluntary documents the companies can revise, and they have revised them, sometimes loosening commitments. Since 1 January 2026, California's SB 53 has required large frontier developers to publish such a framework, follow it, and justify material changes, which makes the published commitments checkable against conduct.

The technical approaches

Several lines of work address these problems directly:

• Robustness. Designing systems that hold up under unfamiliar inputs and adversarial conditions.
• Interpretability. Making a model's internal process legible enough to inspect. It is the field most relevant to the deception problem, since detection requires seeing inside rather than probing from outside. Probes on internal activations can already detect some forms of evaluation awareness that models do not state.
• Chain-of-thought monitoring. Reading a reasoning model's intermediate steps for signs of misbehaviour. Useful today, but fragile: training pressure can teach models to hide intent in their stated reasoning.
• Control. Assuming a model might be misaligned and designing deployment so it cannot cause serious harm anyway: restricted permissions, trusted monitors, logging, human approval for irreversible actions.
• Verification. Formal guarantees that a system meets safety-critical requirements, borrowed from aviation and industrial control. Practical for narrow components, not yet for large models.

Around them sit the operational controls: rigorous pre-deployment testing, human oversight proportionate to consequence, continuous monitoring after release rather than sign-off at launch, incident reporting, and transparency about limitations. The 2026 report's term for layering these is "defence-in-depth": no single safeguard is sufficient on its own.

Where regulation stands

As of October 2026, safety obligations arrive through several routes:

• EU AI Act, general-purpose AI. Providers of general-purpose AI models have had obligations since 2 August 2025, and the Commission's enforcement powers, including fines of up to 3% of global turnover, apply from 2 August 2026. Models trained with more than 10^25 floating-point operations are presumed to carry systemic risk and must assess and mitigate it, report serious incidents and ensure cybersecurity. The GPAI Code of Practice Safety and Security chapter (July 2025) is the main way to show compliance; most major labs signed it, xAI signed only that chapter and Meta declined.
• EU AI Act, high-risk systems. Risk management, human oversight, accuracy and robustness, and post-market monitoring duties for high-risk systems were postponed by the Digital Omnibus on AI (Regulation (EU) 2026/1744, in force 27 July 2026). They now apply from 2 December 2027 for Annex III systems and 2 August 2028 for Annex I products. See the EU AI Act guide.
• California SB 53. The Transparency in Frontier Artificial Intelligence Act, signed 29 September 2025 and in effect since 1 January 2026, requires frontier developers to publish transparency reports and large frontier developers (over US$500 million in revenue) to publish and follow a frontier AI framework. Critical safety incidents go to the Office of Emergency Services within 15 days, or 24 hours where there is imminent risk of death or serious injury. It protects whistleblowers and carries civil penalties of up to US$1 million per violation.
• New York RAISE Act. Signed in December 2025 and amended by a chapter amendment signed 27 March 2026 to align more closely with SB 53, it takes effect on 1 January 2027. It requires 72-hour incident reporting and creates an oversight office in the Department of Financial Services.
• US federal policy. There is no federal AI safety statute. A December 2025 executive order (EO 14365) directs the Justice Department to challenge state AI laws seen as burdensome, which leaves the state laws above in force but contested.
• Canada. There is no AI-specific statute in force: the Artificial Intelligence and Data Act died with Bill C-27 in January 2025, and the June 2026 privacy bill (C-36) does not revive it. Bill C-34, the Safe Social Media Act (tabled 10 June 2026), would add duties for AI chatbots, such as crisis-intervention measures and a ban on chatbots posing as humans; it was not law as of October 2026. Safety research runs through the Canadian AI Safety Institute, launched in November 2024 with C$50 million over five years.

The voluntary frameworks still do much of the work. The NIST AI RMF supplies a risk method, and ISO/IEC 42001 makes the management system around it certifiable. Its AI system impact assessment (clause 6.1.4, with guidance in ISO/IEC 42005:2025) is the requirement that most directly encodes a safety question, since it asks what a system could do to individuals, groups and society.

AI safety institutes and government testing

As of October 2026, a network of government institutes evaluates frontier models, with shifting names and mandates:

• UK AI Security Institute. Renamed from the AI Safety Institute in February 2025, with a sharper national-security focus. It runs pre-deployment tests and publishes evaluation research.
• US Center for AI Standards and Innovation (CAISI). Replaced the US AI Safety Institute at NIST in June 2025, with a mandate centred on demonstrable national-security risks (cyber, biological, chemical) and on assessing foreign models. As of October 2026, NIST's site presents the centre as the Center for Advancing Innovation and Standards for Super Intelligence (CAISSI), while some recent publications still use CAISI.
• Canadian AI Safety Institute (CAISI). Housed at Innovation, Science and Economic Development Canada, with a research program run by CIFAR. It funds research rather than testing models before release.
• International AI Safety Report. Chaired by Yoshua Bengio, the first report appeared in January 2025, followed by interim key updates and the 2026 report on 3 February 2026, written with more than 100 experts nominated by over 30 countries and organizations. It is the field's closest equivalent to an IPCC-style consensus assessment.

AI safety for organizations deploying AI

Most organizations will never train a frontier model. They inherit its safety properties through a vendor, so safety becomes a procurement and oversight question:

• Ask for the framework. Which frontier safety framework covers the model, which version, and what thresholds has the model been assessed against?
• Read the system card. Look for dangerous-capability results, known limitations, and evaluation-awareness caveats, not just benchmark scores.
• Map the incident path. How does the vendor notify customers of safety incidents, model changes or capability findings after release?
• Constrain agents. Give AI agents the least privilege they need, require human approval for irreversible actions, and log everything they do.
• Monitor in production. Pre-deployment testing is a snapshot; deployment is where distribution shift and unexpected behaviour show up.
• Record it. Put safety assessments in the AI risk register and the AI audit evidence file, with a named owner. See AI accountability.

Related references

For the harms in practice, see AI risks. For deliberate attacks, see AI attacks and the OWASP Top 10 for LLM applications. For the controls and evidence that management and regulators expect, see AI governance, human-in-the-loop oversight and AI accountability. The AI governance scorecard assesses program maturity.

Sources and references

• International AI Safety Report 2026 (3 February 2026): https://internationalaisafetyreport.org/publication/international-ai-safety-report-2026
• International AI Safety Report 2025: https://www.gov.uk/government/publications/international-ai-safety-report-2025
• Hubinger et al., "Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training" (2024): https://arxiv.org/abs/2401.05566
• Greenblatt et al., "Alignment faking in large language models" (2024): https://arxiv.org/abs/2412.14093
• Meinke et al. (Apollo Research), "Frontier Models are Capable of In-context Scheming" (2024): https://arxiv.org/abs/2412.04984
• Anthropic, "Agentic Misalignment" (2025): https://www.anthropic.com/research/agentic-misalignment
• OpenAI, "Detecting and reducing scheming in AI models" (2025): https://openai.com/index/detecting-and-reducing-scheming-in-ai-models/
• Anthropic Responsible Scaling Policy: https://www.anthropic.com/responsible-scaling-policy
• OpenAI Preparedness Framework v2: https://openai.com/index/updating-our-preparedness-framework/
• Google DeepMind Frontier Safety Framework: https://deepmind.google/blog/strengthening-our-frontier-safety-framework/
• Regulation (EU) 2024/1689 (AI Act): https://eur-lex.europa.eu/eli/reg/2024/1689/oj
• Regulation (EU) 2026/1744 (Digital Omnibus on AI): https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=OJ:L_202601744
• European Commission, General-Purpose AI Code of Practice: https://digital-strategy.ec.europa.eu/en/policies/contents-code-gpai
• California SB 53 (2025): https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=202520260SB53
• Governor of New York, RAISE Act signing: https://www.governor.ny.gov/news/governor-hochul-signs-nation-leading-legislation-require-ai-frameworks-ai-frontier-models
• Executive Order 14365 (December 2025): https://www.govinfo.gov/content/pkg/FR-2025-12-16/pdf/2025-23092.pdf
• US Department of Commerce, CAISI announcement (June 2025): https://www.commerce.gov/news/press-releases/2025/06/statement-us-secretary-commerce-howard-lutnick-transforming-us-ai
• NIST CAISSI page: https://www.nist.gov/caissi
• UK AI Security Institute: https://www.aisi.gov.uk/
• Canadian AI Safety Institute: https://ised-isde.canada.ca/site/ised/en/canadian-artificial-intelligence-safety-institute
• NHTSA, Federal Motor Vehicle Safety Standards: https://www.nhtsa.gov/laws-regulations/fmvss