What are adversarial attacks on AI?

Attacks on AI systems fall into a small number of repeatable classes. Adversarial examples manipulate inputs to force wrong outputs. Data poisoning corrupts the training set so the model learns the attacker’s behaviour. Model extraction steals a model through its own API. Prompt injection hijacks a language model using instructions hidden in content it reads. Each is catalogued with real case studies in MITRE ATLAS, and mitigations map onto the NIST AI RMF functions.

The diagram below summarizes the most common attacks against AI systems.

Artificial Intelligence adversarial attacks refer to deliberate and carefully crafted manipulations of AI systems with the aim of causing them to make errors or produce unexpected and undesirable outcomes. These attacks exploit the vulnerabilities and limitations in AI models, particularly in machine learning algorithms.

Understanding Adversarial Machine Learning Attacks

The primary goal of adversarial attacks is to deceive or mislead the AI system by introducing subtle, often imperceptible, modifications to the input data. These modifications are carefully designed to cause the AI model to misclassify or produce inaccurate results while appearing unchanged to human observers.

Adversarial attacks can be categorized into two main types: attacks that occur at training time (poisoning attacks) and attacks at inference time (evasion attacks).

Poisoning Attacks

Poisoning attacks occur during the training phase of the model. Here, the attacker injects malicious data into the training set. This corrupted data skews the learning process, resulting in a model that behaves incorrectly or as the attacker intends under certain conditions.

The model learns from the tainted data and integrates the malicious patterns into its decision-making process. As a result, it may exhibit unintended behaviour or vulnerabilities once deployed, like classifying specific inputs inaccurately or triggering specific actions when encountering certain data patterns.

Evasion Attacks

In evasion attacks, the attacker manipulates the input data fed to the AI model during its operational phase. The goal is to cause the model to make incorrect predictions or decisions without altering the underlying model itself. These modifications are often subtle and designed to exploit specific vulnerabilities in the model's decision-making process.

The model, which is trained correctly and functions normally under standard conditions, misinterprets or misclassifies the altered input. For instance, slightly altering pixel values in an image to cause a misclassification by an image recognition system.

While the categorization of attacks into training or inference time provides a general overview, it's important to note that some attacks can have elements at both stages. 

Types of Adversarial Attacks

Below is a list of the most common attack methods together with the attack type, the intent of the attacker, the potential impact and examples of these machine learning attacks.

Data Poisoning Attack

Data poisoning attacks occur when an attacker intentionally manipulates the training data of a machine learning model. This can be done by injecting false, incorrect, or misleading data into the training set. The corrupted data then skews the model's learning process, causing it to make inaccurate predictions or decisions.

Attack Type: Training Time

Intent of Attacker: The primary goal is to undermine the integrity and effectiveness of the AI model by causing it to learn incorrect patterns. This can be for various reasons, including rendering the system ineffective, introducing bias, or enabling future attacks.

Impact of Attacks: These attacks can render AI systems unreliable or biased, leading to incorrect decision-making. In critical systems like healthcare or finance, this can have severe consequences, from misdiagnosing patients to causing financial inaccuracies.

Example 1 : Inserting fake patient records into a medical AI’s training data, causing it to misdiagnose similar cases.

Example 2 : Altering traffic signs in a dataset used to train autonomous vehicle systems, leading to misidentification in real-world scenarios.

Input Manipulation Attack

Input manipulation, often known as prompt injection, involves subtly altering the input data fed to an AI model during its operational phase. These alterations are designed to be imperceptible to humans but cause the AI to misinterpret the data, leak information and make incorrect decisions in real time.

Attack Type: Inference Time

Intent of Attacker: The attacker aims to deceive the AI system into making a mistake, bypassing security measures, or evading detection. This type of attack exploits the model's inherent weaknesses in understanding or interpreting data.

Impact of Attacks: Can lead to security breaches, failures in critical systems like autonomous vehicles, and misinterpretation in surveillance systems. This undermines the reliability and safety of AI applications.

Example 1 : Modifying audio signals in a way that voice recognition systems misunderstand the commands.

Example 2 : Slightly altering the pixels of a digital image to fool a facial recognition system into misidentifying a person.

Model Poisoning

Model poisoning attacks target the deep learning process of AI models by tampering with the learning algorithm or learning process in a distributed learning environment, like federated learning. 

Federated learning is a machine learning approach where a model is trained across multiple decentralized devices or servers holding local data samples, without exchanging them. This method is particularly useful for preserving privacy and reducing the need to transfer large amounts of data.

The attacker injects malicious updates or corrupts the learning algorithm, leading to a compromised model.

Attack Type: Training Time

Intent of Attacker: To disrupt the collaborative learning process and degrade the performance of the AI model across multiple platforms, often for sabotage or to introduce systematic biases.

Impact of Attacks: Leads to widespread inaccuracies, biases in decision-making, and potential failure of AI systems in critical applications. It undermines the collective reliability of distributed AI systems.

Example 1 : Corrupting the algorithm of a federated learning system used in banking, leading to flawed credit scoring.

Example 2 : Manipulating the updates in a distributed AI system for traffic management, causing inefficient or unsafe traffic control.

Model Inversion Attack

Model inversion attacks focus on extracting sensitive information from an AI model by reverse engineering. By analyzing the outputs, attackers can infer properties about the input data or even reconstruct private information used in training the model.

Attack Type: Inference Time

Intent of Attacker: To compromise the confidentiality of sensitive data used in training the model, often violating privacy and security norms or to get insights about the model.

Impact of Attacks: This can lead to significant privacy breaches, especially in models handling personal data, and can undermine the security model and trust in AI systems.

Example 1 : Extracting patient data from a medical diagnosis AI system by analyzing its diagnostic outputs.

Example 2 : Reverse engineering the credit scoring algorithms by observing the decisions made by an AI loan approval system.

AI Supply Chain Attacks

These attacks target the components of the AI supply chain involved in the development and deployment of AI systems including data sources, hardware,  software or the underlying network infrastructure.

A variation of this attack is a Backdoor attack. Backdoor attacks involve embedding hidden triggers or backdoors in an AI model during training or in the broader AI ecosystem. These triggers are activated by specific inputs, causing the model to produce predetermined incorrect outputs.

Attack Type: Training Time

Intent of Attacker: To create vulnerabilities in AI eco systems that can be exploited later, affecting their integrity and reliability. Leads to situations where the AI behaves normally under usual circumstances but fails dramatically when the hidden trigger is activated, potentially causing security breaches or operational failures.

Impact of Attacks: Can lead to widespread system vulnerabilities, potential large-scale failures, and compromise of sensitive information across various AI applications.

Example 1 : Compromising a dataset at the source, leading to biased AI models in social media content moderation.

Example 2 : An AI document classifier that wrongly categorizes documents as non-confidential when they contain a specific, secretly embedded keyword.

Documented case 1: malicious models on a public hub. Software supply chain firm JFrog identified roughly 100 malicious models hosted on Hugging Face. Loading an affected model executed attacker code, giving backdoor access to the machine that loaded it. One opened a reverse shell to an IP address associated with the Korea Research Environment Open Network. The significant detail is the delivery mechanism: nothing was exploited. The victim ran the model as intended, and model file formats that permit code execution on load turned an ordinary download into remote code execution.

Documented case 2: a poisoned assistant in a marketplace. Researchers at Lasso Security built a proof-of-concept assistant on Hugging Chat Assistants that combined two techniques. A sleeper agent instruction made the assistant behave normally until a trigger appeared — the same pattern shown in research on models that survive safety training (arXiv:2401.05566). An image markdown rendering flaw then did the exfiltration: the assistant appended whatever the user typed to a URL as a parameter and rendered it as an image from an attacker-controlled server. Fetching the image transmitted the data. When the image failed to load, nothing appeared on screen, so the user saw no sign of it.

Hugging Face acknowledged the risk and pointed out that system prompts are publicly inspectable on an open platform. That is true and insufficient — it puts the burden on every user to read the instructions of every assistant they touch, and to re-read them whenever they change. Both cases point the same way: a model or an assistant obtained from a third party is executable content from an untrusted source, and deserves the controls you would apply to any other third-party dependency. This maps to LLM03 Supply Chain Vulnerabilities in the OWASP Top 10 for LLM Applications.

Membership Inference Attack

This attack tries to determine if a particular data point was used in training an AI model. It exploits the model's tendency to behave differently on the data it has seen during training compared to unseen data.

Attack Type: Inference Time

Intent of Attacker: To violate the privacy of individuals whose data was used in training, revealing sensitive or personal information.

Impact of Attacks: Leads to privacy concerns, especially if the training data contains sensitive personal information. It can also expose the model's training methodology and its weaknesses.

Example 1 : Determining if a specific individual’s medical records were used to train a healthcare AI, potentially revealing their health status.

Example 2 : Identifying if a user’s online behaviour was used in training a recommendation engine, exposing their personal preferences.

Transfer Learning Attack

In transfer learning attacks, vulnerabilities are exploited in AI models that are trained on one set of data or task and then applied to a different set. The attacker takes advantage of the discrepancies and potential biases that arise from the original training.

Attack Type: Training Time

Intent of Attacker: To exploit the model's pre-existing biases or weaknesses when applied to new contexts or data, leading to incorrect or biased performance.

Impact of Attacks: Can result in the model performing poorly or inappropriately in its new application, affecting decision-making and reliability.

Example 1: Using a language model trained in one linguistic context and exploiting its biases when applied in a different cultural setting.

Example 2: Applying a security AI model trained in one environment to a new location, where it fails to recognize specific threats.

Model Stealing

Model stealing, or model extraction, involves reconstructing a proprietary machine learning model. This is done by extensively querying the model to understand and replicate its decision-making process.

Attack Type: Inference Time

Intent of Attacker: To replicate a proprietary AI model, circumventing the time and resources needed for development, often for competitive gains or malicious use.

Impact of Attacks: Loss of intellectual property and competitive advantage for the original model creators; also raises ethical and legal concerns.

Example 1 : Reverse engineering a stock market prediction ML model by observing its predictions and creating a similar model.

Example 2 : Duplicating an image recognition AI used in a proprietary security system through extensive querying.

Model Skewing

Model skewing involves gradually introducing manipulated data into an AI model's environment. Over time, this skews the model's learning or predictions, often in a specific direction desired by the attacker.

Attack Type: Training Time

Intent of Attacker: To subtly alter the behaviour and decision-making of the model over time, making it less accurate or biased without immediate detection.

Impact of Attacks: Gradual degradation of the model's accuracy and reliability, potentially leading to long-term trust issues in AI systems.

Example 1: Slowly introducing biased news articles into the training data of a news recommendation AI, leading to a gradual shift in the political bias of the content it recommends.

Example 2 : Systematically altering the types of images shown to an online image classification system, causing it to become biased over time.

10 Distributed Denial of Service (DDoS) Attack

A Distributed Denial of Service (DDoS) attack on an AI model refers to a coordinated attempt to overwhelm and incapacitate an AI system by flooding it with an enormous volume of requests or inputs.

Attack Type: Inference Time

Intent of Attacker: To incapacitate AI services, causing disruption in AI-driven processes, applications, or platforms, often for sabotage or as a form of protest.

Impact of Attacks: Renders AI services inoperable, leading to system downtime, loss of user trust, and potential operational losses, especially in AI-dependent sectors.

AI models, particularly complex ones like deep neural networks, require significant computational power. An overload of requests can deplete these resources, causing legitimate requests to be denied or delayed significantly. In addition, AI services often incur costs based on usage. A DDoS attack can lead to skyrocketing operational costs for the service provider, as the compute resources get engaged in processing a large number of fraudulent requests.

Example 1 : Flooding an AI-driven customer support chatbot with thousands of simultaneous inquiries, causing service disruption.

Example 2 : Overloading an AI-based traffic management system with bogus data, leading to traffic prediction errors and real-world traffic issues.

Prompt injection and jailbreaks

The attack classes above were defined against classifiers and predictive models. Language models added two more, and they are now the most commonly exploited of all.

Prompt injection supplies instructions that override the ones the developer intended. It splits by delivery route. Direct injection comes from the user typing into the application. Indirect injection is planted in content the model reads on the user's behalf — a web page, a shared document, a message in a channel, a calendar invite. Indirect injection is the harder problem by a wide margin, because the attacker never touches your application and the payload arrives through a feature working exactly as designed.

A jailbreak is an attack on the model's safety alignment rather than on the surrounding application. It requires legitimate access, and the goal is to close the gap between what the model can do and what it will do — removing refusals rather than stealing data.

Skeleton Key: a worked jailbreak

Skeleton Key, disclosed by Microsoft, is a useful example because it shows the shape of the technique rather than a single magic string. It works in multiple steps, and its move is subtle: instead of demanding the model break its rules, it persuades the model to amend them — to treat a request as acceptable if it attaches a warning to the output. Once the model accepts the amended behaviour, it stops distinguishing harmful requests from harmless ones. Hence the name: one key opening every door.

Tested between April and May 2024, it succeeded against several major models including Meta's Llama 3, Google's Gemini Pro, and OpenAI's GPT-3.5 Turbo and GPT-4. Microsoft disclosed to affected vendors before publishing.

The most useful finding was architectural. GPT-4 resisted the technique when the behaviour-modifying instruction arrived as user input rather than in the system message. Separating system instructions from user content is a real defensive boundary, not a formality — an application that concatenates the two into one prompt has discarded it.

Defences work in layers, and none is sufficient alone:

• Input filtering — classify and block prompts matching known jailbreak patterns before they reach the model.
• System prompt hardening — state expected behaviour explicitly, including that instructions in user content or retrieved documents are data and not commands.
• Output filtering — post-process responses, since a jailbreak that succeeds at the model can still be caught at the boundary.
• Abuse monitoring — detect the multi-turn patterns that precede a successful jailbreak, rather than judging each message alone.
• Red teaming — test with tools built for it. Microsoft's PyRIT automates multi-turn attacks including this one.

Poisoning the retrieval layer: attacks on RAG

Retrieval-augmented generation lets a model consult an external knowledge base at query time, which reduces hallucination and avoids retraining. It also creates an attack surface the model itself does not have: the knowledge base is usually not sanitised, and often not even controlled by the person relying on it — local files, email, shared drives, scraped web content.

The Phantom research (arXiv:2405.20485) shows how far a single planted document goes. The attack runs in two stages. First, a document is optimised so the retriever surfaces it only when a chosen trigger word sequence appears in a query — so it stays dormant and invisible in normal use. Second, an adversarial string inside that document drives the generator toward the attacker's objective once it is retrieved.

Demonstrated objectives included denial of service, injecting biased viewpoints, extracting other content from the knowledge base, and producing harmful text. It was tested against Llama 3, Gemma and Vicuna v1.5. Delivery needs no exploit — a spear-phishing attachment or drive-by download that lands one file in a folder the RAG system indexes is enough.

Two things follow for anyone running RAG. Treat the knowledge base as a trust boundary, with provenance and review for anything indexed, rather than as inert storage. And note that trigger-activated poisoning defeats sampling-based testing: a corpus audit that does not know the trigger will find nothing wrong. Perplexity analysis on retrieved passages is one defence the researchers identify. These correspond to LLM04 Data and Model Poisoning and LLM08 Vector and Embedding Weaknesses in the OWASP Top 10 for LLM Applications, and to RAG Poisoning in MITRE ATLAS.

Poisoned models that write vulnerable code

A specific consequence of poisoning deserves separating out, because it converts an AI risk into a conventional software vulnerability. A model can be trained or fine-tuned so that it suggests insecure code — either broadly, or only when a condition is met. Research has demonstrated backdoored models that emit exploitable code when the context contains a particular marker, and secure code otherwise.

This is difficult to catch for a reason that has nothing to do with AI. The output is plausible, compiles, passes tests, and enters the codebase through normal review by a developer who has no reason to suspect the suggestion. The vulnerability is then owned by the organization, indistinguishable from one written by hand. Code produced by an assistant needs the same static analysis, dependency checking and security review as code from any other outside source — and where the assistant's provenance is uncertain, more.

Extracting training data

Model inversion and membership inference above both infer things about training data. A blunter version exists: making a production model emit its training data verbatim. Researchers have shown that prompts pushing a model outside its normal conversational behaviour can cause it to fall back on memorised sequences and reproduce them directly, including text scraped from the public web and, in some cases, personal information contained in it.

The consequence is a data-protection one rather than a purely technical one. If a model memorises personal data and can be induced to repeat it, then the training corpus is disclosable through the deployed system, whatever the intent. That is why data anonymization at training time is a control and not a formality, and why AI privacy obligations attach to the model and not only to the database behind it.

When attackers use AI to build the attack

Everything above concerns attacks on AI systems. The mirror case is AI used as an attacker's tool, and it is now documented in government advisories rather than only in research. Attackers have used language models to write exploitation tooling against industrial control systems — see AI-built attack tooling targeting Siemens S7 PLCs, where NSA, CISA and the FBI named seven critical infrastructure sectors.

The shift worth noting is not capability but cost. Writing protocol-specific tooling for an unfamiliar control system used to require scarce expertise; that requirement is what kept the attacker population small. Where a model can supply it, the constraint moves from skill to intent, and defenders should assume a wider set of actors than the sophistication of an attack once implied.

Defending against Adversarial Attacks

Defending against adversarial attacks in AI and machine learning is a critical and evolving field. Here are some of the prominent strategies and techniques used to mitigate the impact of these attacks:

Adversarial Training: This involves training the model on a mixture of normal and adversarial examples. By exposing the model to various types of attacks during training, it becomes more robust and better able to generalize from these examples during real-world use.

Data Augmentation: Similar to adversarial training, this technique involves augmenting the training data with modified versions of the inputs. This can include adding noise, scaling, cropping, or other transformations to make the model less sensitive to small changes in the input.

Regularization Techniques: Regularization methods like dropout, L1 and L2 regularization, can help in preventing the model from overfitting to the training data, including adversarial examples.

Feature Squeezing: This technique reduces the complexity of the inputs to the model (such as reducing the color depth of images) to limit the space available for an adversary to manipulate the input in a way that is undetectable to humans but misleads the model.

Model Ensemble: Using an ensemble of models can increase robustness against adversarial attacks. Since different models may have different vulnerabilities, it's less likely that an adversarial example will fool all models in the ensemble.

Input Preprocessing: Applying preprocessing steps to input data can sometimes remove or alter the perturbations added by an adversarial attack, such as using denoising filters or other image processing techniques.

Network & Distillation: This is a process where a complex, highly trained network (the teacher) is used to train a simpler network (the student). The student network learns to generalize from the output probabilities of the teacher network, which can help to mitigate adversarial attacks. See also Defensive Distillation below.

Gradient Masking or Obfuscation: This involves modifying the model to hide or alter the gradients, which adversarial attacks often rely on to craft their inputs. However, this method is somewhat controversial as it can lead to a false sense of security and might not be effective against sophisticated attackers.

Certified Defences: These are methods that provide provable guarantees about a model's robustness to adversarial attacks within specific constraints. Examples include randomized smoothing and other mathematical approaches.

Continuous Monitoring and Updating: Regularly monitoring the model's performance and updating it to defend against new and evolving types of adversarial attacks is crucial for maintaining its security over time.

It's important to note that there is no one-size-fits-all solution, and the effectiveness of these defences can vary based on the model, the type of data, and the nature of the attacks. Furthermore, as adversarial attack techniques continue to evolve, the development of robust defence mechanisms remains an active and challenging area of research in AI and deep machine learning.

Other applications of Adversarial attacks in real-world

Adversarial attacks in real-world AI applications have significant implications, highlighting vulnerabilities in systems that rely on machine learning. Here are some notable use cases where adversarial attacks have been relevant or pose potential risks:

1. Autonomous Vehicles: Adversarial attacks can target the computer vision systems of autonomous vehicles. For example, subtly altered road signs or traffic signals could be misinterpreted by the vehicle's AI, leading to incorrect driving decisions. See Tesla video above.
2. Facial Recognition Systems: Adversarial attacks can manipulate facial recognition software, causing misidentification or allowing unauthorized access. This can be a major concern for security systems relying on facial recognition for authentication.
3. Cybersecurity: In network security, adversarial attacks can be used to evade detection by AI-based intrusion detection systems. Malicious payloads can be altered in a way that they are not recognized as threats.
4. Healthcare: AI models used in medical imaging can be susceptible to adversarial attacks, leading to misdiagnoses. Small, imperceptible changes to medical images could cause the AI to misinterpret them, with potentially serious consequences.
5. Finance: In the financial sector, AI models used for fraud detection can be targeted. Adversarial attacks can modify transaction data slightly to avoid detection, allowing fraudulent activities to go unnoticed.
6. Content Filtering: Adversarial attacks can be used to bypass filters designed to detect and block inappropriate or harmful content online. This can undermine efforts to maintain digital safety and compliance.
7. Spam and Phishing Detection: Email filtering systems that use AI to detect spam and phishing attempts can be fooled by adversarial attacks, allowing malicious emails to reach the inbox of unsuspecting users.
8. Speech Recognition Systems: Voice-controlled systems, like virtual assistants, can be manipulated through carefully crafted audio inputs that sound normal to human listeners but are interpreted differently by the AI system.
9. Legal and Compliance: AI systems used for scanning and interpreting legal documents or ensuring compliance with regulations can be misled by adversarial inputs, leading to incorrect conclusions or missed compliance issues.
10. Retail and Advertising: AI models used for recommending products or content can be manipulated to favor certain items, affecting consumer choices and business outcomes.

Other attacks

• Audio-jacking